Oct 30 02:40:21.870704 ip-10-0-128-214 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:40:22.256523 ip-10-0-128-214 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:22.256523 ip-10-0-128-214 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:40:22.256523 ip-10-0-128-214 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:22.257267 ip-10-0-128-214 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:22.259822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259728    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:22.259822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259816    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:22.259822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259822    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:22.259822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259826    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259831    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259836    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259840    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259844    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259848    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259852    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259856    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259860    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259864    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259868    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259873    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259877    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259881    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259885    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259889    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259893    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259897    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259901    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259906    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:22.260075 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259910    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259914    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259918    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259922    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259935    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259941    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259945    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259950    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259954    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259959    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259964    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259968    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259973    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259977    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259982    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259987    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259991    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.259997    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260002    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260007    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:22.260925 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260011    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260016    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260022    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260030    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260037    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260043    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260048    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260053    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260058    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260062    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260067    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260071    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260076    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260080    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260085    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260090    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260094    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260099    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260104    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:22.261769 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260111    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260118    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260125    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260130    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260135    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260140    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260145    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260149    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260154    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260159    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260166    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260170    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260175    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260180    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260184    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260188    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260194    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260199    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260203    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:22.262585 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260208    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260213    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260218    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260223    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260228    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260232    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260237    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260242    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260246    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260251    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260256    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260261    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260267    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260274    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260279    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260284    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260289    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260293    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260298    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260303    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:22.263181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260307    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260312    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260316    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260321    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.260325    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261617    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261630    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261636    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261641    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261645    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261649    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261654    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261659    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261664    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261668    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261673    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261678    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261682    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261687    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261692    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:22.263722 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261698    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261702    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261707    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261716    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261722    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261727    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261732    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261738    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261742    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261747    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261754    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261760    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261765    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261770    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261774    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261779    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261783    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261788    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261792    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261797    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:22.264592 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261801    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261806    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261811    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261816    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261820    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261825    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261830    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261834    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261838    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261843    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261848    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261853    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261857    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261863    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261867    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261872    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261877    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261883    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261888    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261892    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:22.265523 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261897    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261901    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261906    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261910    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261915    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261919    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261926    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261930    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261936    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261940    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261945    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261949    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261954    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261958    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261964    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261969    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261974    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261978    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261983    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261988    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:22.266391 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261992    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.261997    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262002    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262006    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262011    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262016    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262021    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262025    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262031    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262036    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262041    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262046    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262051    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262056    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262061    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262066    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262070    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262075    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262079    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:22.267154 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262084    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262090    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262095    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262099    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262104    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262109    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262113    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262118    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262122    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262126    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262131    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.262136    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263843    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263854    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263860    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263866    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263871    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263877    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263882    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:22.267822 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263887    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263894    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263903    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263910    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263915    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263920    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263924    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263929    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263934    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263938    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263943    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263948    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263953    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263958    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263963    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263967    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263972    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263976    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263981    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:22.268369 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263986    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263990    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.263995    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264000    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264004    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264009    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264014    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264018    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264023    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264028    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264033    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264038    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264043    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264047    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264054    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264059    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264064    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264069    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264073    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264078    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:22.268871 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264083    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264087    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264092    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264096    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264101    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264105    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264109    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264114    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264118    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264123    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264128    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264132    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264137    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264142    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264146    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264150    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264155    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264160    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264164    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264171    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:22.269443 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264175    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264180    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264184    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264188    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264193    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264198    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264204    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264209    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264213    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264218    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264225    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264230    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264235    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264239    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264244    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264248    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264252    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264257    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264262    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264266    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:22.269941 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264271    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264276    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264280    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264285    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264289    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264294    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264299    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264304    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264309    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264313    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264318    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264326    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264331    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264335    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264343    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264350    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264357    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264363    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264370    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:22.270411 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.264375    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264531    2571 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264552    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264562    2571 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264571    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264579    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264584    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264592    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264599    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264604    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264609    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264615    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264621    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264627    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264632    2571 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264637    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264642    2571 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264647    2571 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264652    2571 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264661    2571 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264666    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264672    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264677    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264683    2571 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:40:22.270891 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264693    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264698    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264704    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264709    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264715    2571 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264720    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264725    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264731    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264736    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264743    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264750    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264755    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264761    2571 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264767    2571 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264772    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264779    2571 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264784    2571 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264790    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264795    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264801    2571 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264808    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264814    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264819    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264824    2571 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264830    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:40:22.271467 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264835    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264840    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264845    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264850    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264855    2571 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264860    2571 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264867    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264876    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264883    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264890    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264895    2571 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264900    2571 flags.go:64] FLAG: --help="false"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264906    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264911    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264916    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264921    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264927    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264933    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264939    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264944    2571 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264949    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264955    2571 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264960    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264966    2571 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:40:22.272190 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264971    2571 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264977    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264982    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264987    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264992    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.264997    2571 flags.go:64] FLAG: --lock-file=""
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265002    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265007    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265013    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265023    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265028    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265033    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265038    2571 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265043    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265050    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265056    2571 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265062    2571 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265069    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265074    2571 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265081    2571 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265087    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265092    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265097    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265102    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265108    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:40:22.272803 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265113    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265118    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265130    2571 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265136    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265141    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265148    2571 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265153    2571 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265158    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265163    2571 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265168    2571 flags.go:64] FLAG: --port="10250"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265174    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265179    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0231e2d94c84776d2"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265184    2571 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265190    2571 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265195    2571 flags.go:64] FLAG: --register-node="true"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265200    2571 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265207    2571 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265212    2571 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265218    2571 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265223    2571 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265230    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265236    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265242    2571 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265247    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265252    2571 flags.go:64] FLAG: --runonce="false"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265257    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:40:22.273495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265263    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265268    2571 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265273    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265278    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265284    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265289    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265294    2571 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265299    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265305    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265310    2571 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265315    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265320    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265327    2571 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265332    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265341    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265347    2571 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265352    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265359    2571 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265364    2571 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265369    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265374    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265379    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265385    2571 flags.go:64] FLAG: --v="2"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265393    2571 flags.go:64] FLAG: --version="false"
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265401    2571 flags.go:64] FLAG: --vmodule=""
Oct 30 02:40:22.274101 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265408    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.265413    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265689    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265699    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265704    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265709    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265713    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265718    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265723    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265728    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265733    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265738    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265743    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265747    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265752    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265756    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265763    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265771    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265776    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:22.274729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265781    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265787    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265791    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265796    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265800    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265805    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265809    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265814    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265819    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265824    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265828    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265833    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265837    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265841    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265846    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265853    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265860    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265866    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265871    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:22.275181 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265876    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265880    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265885    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265890    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265895    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265900    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265905    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265909    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265914    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265918    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265923    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265927    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265932    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265937    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265941    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265947    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265952    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265956    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265960    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265965    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:22.275700 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265970    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265975    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265979    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265984    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265989    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265994    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.265998    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266003    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266009    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266014    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266019    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266024    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266028    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266033    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266037    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266042    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266046    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266051    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266056    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266060    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:22.276169 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266065    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266070    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266075    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266079    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266084    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266088    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266093    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266098    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266104    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266109    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266113    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266118    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266123    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266127    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266132    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266136    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266140    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266145    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266150    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:22.276693 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266154    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266160    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266165    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266170    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266174    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266179    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266184    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266189    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266193    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266198    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266203    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.266244    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266306    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266313    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266319    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266324    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:22.277132 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266329    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266334    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266338    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266343    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266347    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266351    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266357    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266362    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266367    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266371    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266376    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266380    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266385    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266389    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266394    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266398    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266403    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266410    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266415    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:22.277529 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266437    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266442    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266447    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266452    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266456    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266461    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266466    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266471    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266475    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266481    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266486    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266491    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266496    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266501    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266505    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266511    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266516    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266520    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266525    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266530    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:22.277974 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266536    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266541    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266546    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266550    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266555    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266560    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266564    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266569    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266574    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266579    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266585    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266590    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266597    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266601    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266606    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266611    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266617    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266622    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266626    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:22.278465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266631    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266637    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266642    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266647    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266652    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266656    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266661    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266666    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266670    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266675    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266679    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266684    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266689    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266694    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266699    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266704    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266709    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266714    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266719    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:22.278918 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266725    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266733    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266738    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266743    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266750    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266755    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266761    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266766    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266770    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266775    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266780    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266785    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266790    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266795    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266799    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266804    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266809    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266813    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266818    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:22.279360 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266825    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266831    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266838    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266843    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266847    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.266851    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.273985    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.274002    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.274746    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.274756    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274851    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274856    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274861    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274864    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274867    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274870    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274873    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274876    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274878    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274881    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:22.279839 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274884    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274887    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274890    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274892    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274895    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274898    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274900    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274903    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274905    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274908    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274911    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274913    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274916    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274918    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274921    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274924    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274926    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274929    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274931    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274935    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:22.280311 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274937    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274946    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274949    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274951    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274954    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274957    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274959    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274962    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274964    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274968    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274972    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274975    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274977    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274980    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274983    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274986    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274988    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274991    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274993    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274996    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:22.280806 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.274999    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275001    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275004    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275006    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275009    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275012    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275014    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275017    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275020    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275022    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275025    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275028    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275031    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275034    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275042    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275045    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275047    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275050    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275053    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:22.281269 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275055    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275058    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275060    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275063    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275065    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275068    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275070    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275073    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275076    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275078    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275081    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275083    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275086    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275088    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275091    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275094    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275096    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275099    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275101    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275104    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:22.281729 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275106    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275110    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275114    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275117    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275120    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275123    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275125    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275128    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275136    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275138    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275141    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275143    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275146    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275148    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275151    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275153    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:22.282232 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275156    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.275186    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275394    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275399    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275403    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275406    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275408    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275411    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275413    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275416    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275418    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275435    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275439    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275442    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275444    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275447    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:22.282630 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275449    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275452    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275454    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275457    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275460    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275463    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275465    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275467    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275470    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275488    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275491    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275494    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275496    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275499    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275501    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275504    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275507    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275509    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275512    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:22.283035 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275514    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275517    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275520    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275522    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275525    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275527    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275529    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275532    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275535    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275537    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275540    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275543    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275545    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275548    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275550    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275553    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275556    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275558    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275561    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275564    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:22.283586 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275566    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275569    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275571    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275579    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275582    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275585    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275587    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275591    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275595    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275599    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275602    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275605    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275608    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275610    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275612    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275615    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275617    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275620    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275622    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:22.284180 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275624    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275627    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275630    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275632    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275635    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275637    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275640    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275642    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275645    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275648    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275651    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275653    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275656    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275658    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275660    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275663    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275666    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275674    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275677    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275679    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:22.284740 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275682    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275684    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275687    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275689    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275692    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275694    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275697    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275699    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275702    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275704    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275707    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275710    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275712    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275715    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.275740    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275786    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:22.285273 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275791    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275793    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275796    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275799    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275801    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275804    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275806    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275809    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275812    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275814    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275817    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275819    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275822    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275824    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275832    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275835    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275837    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275840    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275843    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275845    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:22.285681 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275847    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275850    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275853    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275855    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275858    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275860    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275863    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275865    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275868    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275870    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275875    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275877    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275880    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275883    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275885    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275888    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275891    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275893    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:22.286191 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275897    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275901    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275905    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275908    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275911    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275913    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275916    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275919    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275921    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275930    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275932    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275935    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275937    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275940    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275942    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275945    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275948    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275950    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275953    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275955    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:22.286879 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275958    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275960    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275963    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275965    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275970    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275972    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275975    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275978    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275980    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275983    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275985    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275988    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275991    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275994    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275996    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.275998    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276001    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276004    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276006    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276008    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:22.287359 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276011    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276013    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276021    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276024    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276027    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276029    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276032    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276034    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276037    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276039    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276042    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276044    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276047    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276049    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276052    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276054    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276058    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276060    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276063    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276065    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:22.287927 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276068    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276070    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276073    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276077    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276079    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276082    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:22.276084    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.276132    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.278081    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.279025    2571 server.go:1014] "Starting client certificate rotation"
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.279120    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.279156    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.281056    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:22.288409 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.284820    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:22.293762 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.293742    2571 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:40:22.296746 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.296733    2571 log.go:25] "Validated CRI v1 image API"
Oct 30 02:40:22.297548 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.297526    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:40:22.301224 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.301201    2571 fs.go:135] Filesystem UUIDs: map[7723a7df-d15a-493d-8b60-d7065cdb0280:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a8a23d16-dbad-43b6-8e3f-2b627a766078:/dev/nvme0n1p3]
Oct 30 02:40:22.301267 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.301224    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 02:40:22.306989 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.306883    2571 manager.go:217] Machine: {Timestamp:2026-10-30 02:40:22.305356477 +0000 UTC m=+0.325675704 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104306 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2476f7ae0076573aebef590271bb42 SystemUUID:ec2476f7-ae00-7657-3aeb-ef590271bb42 BootID:5c3f52f6-e09c-4a1e-a410-6c48d533fbb8 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c9:b4:e6:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c9:b4:e6:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:4f:62:b1:ae:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:40:22.307532 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.307519    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:40:22.307623 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.307610    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:40:22.309881 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.309861    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:40:22.310016 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.309883    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-214.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:40:22.310064 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.310019    2571 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:40:22.310064 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.310027    2571 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:40:22.310064 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.310060    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:22.310140 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.310077    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:22.310914 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.310904    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:40:22.314753 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.314742    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:40:22.314875 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.314866    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:40:22.315543 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.315534    2571 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:40:22.315602 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.315547    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:40:22.315602 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.315558    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:40:22.315602 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.315566    2571 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:40:22.315602 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.315574    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:40:22.315953 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.315940    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:22.318092 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.318074    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:40:22.324626 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.324609    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:22.325512 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.325499    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:40:22.325560 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.325530    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:40:22.325679 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.325670    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:40:22.325708 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.325685    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:40:22.325708 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.325691    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:40:22.325708 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.325696    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:40:22.325708 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.325703    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:40:22.325708 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.325709    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:40:22.326175 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.326166    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:40:22.326203 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.326176    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:40:22.326203 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.326184    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:40:22.326203 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.326190    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:40:22.326203 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.326198    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:40:22.326398 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.326206    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:40:22.326822 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.326811    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:40:22.326861 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.326825    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:40:22.330744 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.330730    2571 server.go:1263] "Started kubelet"
Oct 30 02:40:22.330871 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.330824    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:40:22.330940 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.330891    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:40:22.331016 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.330994    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:40:22.331534 ip-10-0-128-214 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:40:22.333316 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.331977    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:40:22.333316 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.332967    2571 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:40:22.337061 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.337038    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 02:40:22.337139 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.337051    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:40:22.337139 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.337067    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:40:22.337237 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.337155    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:22.337655 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.337625    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:40:22.338133 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.337698    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:40:22.338225 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.338136    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:40:22.338225 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.338180    2571 factory.go:55] Registering systemd factory
Oct 30 02:40:22.338225 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.338204    2571 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:40:22.338335 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.338247    2571 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:40:22.338335 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.338255    2571 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:40:22.338507 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.338496    2571 factory.go:153] Registering CRI-O factory
Oct 30 02:40:22.338507 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.338508    2571 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:40:22.338605 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.338573    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:40:22.338605 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.338591    2571 factory.go:103] Registering Raw factory
Oct 30 02:40:22.338605 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.338604    2571 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:40:22.339513 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.339500    2571 manager.go:319] Starting recovery of all containers
Oct 30 02:40:22.351825 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.351806    2571 manager.go:324] Recovery completed
Oct 30 02:40:22.356251 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.356236    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:22.358345 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.358318    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:22.358418 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.358363    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:22.358418 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.358376    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:22.358901 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.358886    2571 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:40:22.359067 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.358898    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:40:22.359131 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.358931    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:22.359131 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.359083    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:40:22.359864 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.359825    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-214.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 02:40:22.360041 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.359990    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:22.360041 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.360015    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:22.360041 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.360031    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:22.361990 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.361350    2571 policy_none.go:50] "Start"
Oct 30 02:40:22.361990 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.361361    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:40:22.361990 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.361370    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:40:22.363492 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.363466    2571 policy_none.go:44] "Start"
Oct 30 02:40:22.371526 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.371509    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:40:22.371602 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.371556    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-62cl7"
Oct 30 02:40:22.371984 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.370987    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-214.ec2.internal.18e32e55b8c65b8b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-214.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-214.ec2.internal,},FirstTimestamp:2026-10-30 02:40:22.330694539 +0000 UTC m=+0.351013765,LastTimestamp:2026-10-30 02:40:22.330694539 +0000 UTC m=+0.351013765,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-214.ec2.internal,}"
Oct 30 02:40:22.378809 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.378790    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-62cl7"
Oct 30 02:40:22.379794 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.379769    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-214.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 02:40:22.380299 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.380270    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 02:40:22.380382 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.380273    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-214.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 02:40:22.380382 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.380272    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 02:40:22.392029 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.391991    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:40:22.396536 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.396524    2571 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:40:22.396608 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.396549    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:40:22.396608 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.396558    2571 server.go:87] "Starting device plugin registration server"
Oct 30 02:40:22.396797 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.396784    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:40:22.396838 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.396798    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:40:22.396908 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.396888    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:40:22.397023 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.396965    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:40:22.397023 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.396985    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:40:22.397574 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.397555    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:40:22.397653 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.397605    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:22.469469 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.469321    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:40:22.471510 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.470534    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:40:22.471510 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.470561    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:40:22.471510 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.470583    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:40:22.471510 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.470624    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 02:40:22.482939 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.482919    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:22.497738 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.497723    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:22.498519 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.498505    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:22.498598 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.498532    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:22.498598 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.498544    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:22.498598 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.498574    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.526181 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.526145    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.526181 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.526164    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-214.ec2.internal\": node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:22.562360 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.562339    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:22.571416 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.571394    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"]
Oct 30 02:40:22.571495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.571458    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:22.572145 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.572133    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:22.572202 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.572157    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:22.572202 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.572166    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:22.574383 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.574371    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:22.574544 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.574530    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.574604 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.574557    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:22.575107 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.575091    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:22.575194 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.575123    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:22.575194 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.575138    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:22.575194 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.575144    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:22.575194 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.575162    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:22.575194 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.575170    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:22.577195 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.577178    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.577275 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.577226    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:22.577873 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.577858    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:22.577937 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.577890    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:22.577937 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.577905    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:22.603008 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.602987    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-214.ec2.internal\" not found" node="ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.605774 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.605753    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-214.ec2.internal\" not found" node="ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.639814 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.639791    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b5a364f76591020026198c5396db417-config\") pod \"kube-apiserver-proxy-ip-10-0-128-214.ec2.internal\" (UID: \"4b5a364f76591020026198c5396db417\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.639878 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.639819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.639878 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.639835    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.663412 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.663394    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:22.740001 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.739982    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.740064 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.740006    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.740064 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.740025    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b5a364f76591020026198c5396db417-config\") pod \"kube-apiserver-proxy-ip-10-0-128-214.ec2.internal\" (UID: \"4b5a364f76591020026198c5396db417\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.740128 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.740068    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b5a364f76591020026198c5396db417-config\") pod \"kube-apiserver-proxy-ip-10-0-128-214.ec2.internal\" (UID: \"4b5a364f76591020026198c5396db417\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.740128 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.740081    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.740128 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.740088    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.764228 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.764209    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:22.864337 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.864259    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:22.906477 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.906456    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.908584 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:22.908567    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:22.965367 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:22.965343    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:23.065872 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:23.065840    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:23.166364 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:23.166342    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:23.257040 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.257014    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:23.267122 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:23.267098    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:23.279216 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.279199    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 02:40:23.279308 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.279293    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:40:23.279361 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.279340    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:40:23.376009 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:23.375982    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:23.383844 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.383809    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:35:22 +0000 UTC" deadline="2028-05-06 03:20:25.304095676 +0000 UTC"
Oct 30 02:40:23.383844 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.383837    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13296h40m1.920261623s"
Oct 30 02:40:23.467656 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:23.467599    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5a364f76591020026198c5396db417.slice/crio-4537d2e4bc34fbe8df163fe61996e315a55601a06ae27833e882a657ca1b7c19 WatchSource:0}: Error finding container 4537d2e4bc34fbe8df163fe61996e315a55601a06ae27833e882a657ca1b7c19: Status 404 returned error can't find the container with id 4537d2e4bc34fbe8df163fe61996e315a55601a06ae27833e882a657ca1b7c19
Oct 30 02:40:23.473310 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.473267    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal" event={"ID":"4b5a364f76591020026198c5396db417","Type":"ContainerStarted","Data":"4537d2e4bc34fbe8df163fe61996e315a55601a06ae27833e882a657ca1b7c19"}
Oct 30 02:40:23.476314 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:23.476297    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:23.477219 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.477205    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:40:23.482465 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:23.482446    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72498d2be80a2b536a0b73a7eab91abc.slice/crio-041e79e9268d06e9f6c0ea23703be9d2adfc7838fe0f0332b090d7d15734ad12 WatchSource:0}: Error finding container 041e79e9268d06e9f6c0ea23703be9d2adfc7838fe0f0332b090d7d15734ad12: Status 404 returned error can't find the container with id 041e79e9268d06e9f6c0ea23703be9d2adfc7838fe0f0332b090d7d15734ad12
Oct 30 02:40:23.494027 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.494015    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:23.495174 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.495160    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:23.495217 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.495191    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:23.495217 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.495205    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:23.577135 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:23.577104    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:23.673553 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.673532    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:23.677606 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:23.677591    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:23.778040 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:23.778015    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:40:23.844837 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.844804    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:23.937515 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.937483    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:23.959402 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.959295    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:23.960613 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.960589    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:40:23.980250 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:23.980225    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:24.316971 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.316943    2571 apiserver.go:52] "Watching apiserver"
Oct 30 02:40:24.338540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.338513    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 02:40:24.361950 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.361920    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:40:24.362597 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.362269    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-wttwd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal","openshift-multus/network-metrics-daemon-d9gmb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr","openshift-multus/multus-additional-cni-plugins-snstr","openshift-multus/multus-nlhw8","openshift-network-diagnostics/network-check-target-k684d","openshift-network-operator/iptables-alerter-tdp5t","kube-system/konnectivity-agent-crnh2","openshift-image-registry/node-ca-28mns","openshift-ovn-kubernetes/ovnkube-node-lxf8j","kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal","openshift-cluster-node-tuning-operator/tuned-ds5x7","openshift-dns/node-resolver-76z2q"]
Oct 30 02:40:24.364995 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.364963    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:24.365106 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.365057    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:24.367129 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.367104    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.369319 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.369298    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:24.369458 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.369373    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:24.371711 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.371691    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.373984 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.373960    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.376217 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.376197    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.378884 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.378862    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:24.383452 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.381946    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:24.383452 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.382048    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:24.385358 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.385340    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.390176 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.390158    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.392297 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392275    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:40:24.392297 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392288    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:40:24.392455 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392416    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hhdzd\""
Oct 30 02:40:24.392505 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392477    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5k7wg\""
Oct 30 02:40:24.392505 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392483    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gf5d6\""
Oct 30 02:40:24.392582 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392560    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:40:24.392631 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392609    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:40:24.392664 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392649    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:40:24.392701 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392652    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:24.392701 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392670    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:40:24.392701 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392649    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:40:24.392804 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392707    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:24.392804 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392742    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ph87b\""
Oct 30 02:40:24.392880 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392800    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:40:24.392880 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.392834    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:40:24.393286 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.393269    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.394267 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.394247    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:40:24.394643 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.394625    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:40:24.394736 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.394631    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4pml\""
Oct 30 02:40:24.395502 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.395485    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.416129 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.416109    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:40:24.416216 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.416178    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jn9vb"
Oct 30 02:40:24.425344 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.425322    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jn9vb"
Oct 30 02:40:24.439524 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.439505    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:40:24.448694 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448670    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-kubelet\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.448797 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448706    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6g6\" (UniqueName: \"kubernetes.io/projected/702224cc-b73b-468d-82e4-1647167c6314-kube-api-access-wg6g6\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.448797 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448739    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-netd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.448797 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7schg\" (UniqueName: \"kubernetes.io/projected/54cc321a-a26d-4aab-8e95-144a2f9aaa41-kube-api-access-7schg\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.448797 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448788    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-sys\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.448966 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448804    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncl2\" (UniqueName: \"kubernetes.io/projected/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-kube-api-access-gncl2\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.448966 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448838    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cnibin\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.448966 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448886    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.448966 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448910    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-kubelet\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.448966 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448933    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-log-socket\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.448966 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448951    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-env-overrides\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.449246 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.448966    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:24.449246 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449022    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm7m6\" (UniqueName: \"kubernetes.io/projected/5f82d4e3-4636-4d13-9770-ae32b7719a7b-kube-api-access-tm7m6\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.449246 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449070    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-cnibin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.449246 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449128    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.449246 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449179    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01973b42-4064-4c7c-b6ef-b85d89b2f152-hosts-file\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.449246 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-etc-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.449569 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449258    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.449569 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449318    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-systemd\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.449569 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-system-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.449569 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449373    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-cni-binary-copy\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.449924 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449904    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:24.449969 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.449948    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.450025 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-netns\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.450080 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450041    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:24.450080 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450060    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-systemd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.450177 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450087    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-script-lib\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.450177 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450152    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/729ae77b-bc29-4685-b33b-38feb7fe0a68-konnectivity-ca\") pod \"konnectivity-agent-crnh2\" (UID: \"729ae77b-bc29-4685-b33b-38feb7fe0a68\") " pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:24.450251 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450189    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-host\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.450251 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450214    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-modprobe-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.450251 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450229    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-multus-daemon-config\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.450251 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450245    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-etc-kubernetes\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.450421 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450293    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01973b42-4064-4c7c-b6ef-b85d89b2f152-tmp-dir\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.450421 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-node-log\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.450421 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450331    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-socket-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.450421 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450369    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/729ae77b-bc29-4685-b33b-38feb7fe0a68-agent-certs\") pod \"konnectivity-agent-crnh2\" (UID: \"729ae77b-bc29-4685-b33b-38feb7fe0a68\") " pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:24.450421 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450397    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysconfig\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.450421 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450419    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-lib-modules\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.450682 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450465    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-bin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.450682 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450488    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-systemd-units\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.450682 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450503    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-var-lib-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.450682 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450546    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-sys-fs\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.450682 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450582    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54cc321a-a26d-4aab-8e95-144a2f9aaa41-host\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.450682 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.450682 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450657    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-multus\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450688    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-ovn\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450713    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450798    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450842    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-config\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450874    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-registration-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450897    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bldfr\" (UniqueName: \"kubernetes.io/projected/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kube-api-access-bldfr\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450920    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-tuned\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450940    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-system-cni-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450978    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.450999    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54cc321a-a26d-4aab-8e95-144a2f9aaa41-serviceca\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.451022 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451013    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15a18601-beef-4a52-a98f-47414f15efe1-iptables-alerter-script\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-bin\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451092    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpg5c\" (UniqueName: \"kubernetes.io/projected/15a18601-beef-4a52-a98f-47414f15efe1-kube-api-access-cpg5c\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451119    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15a18601-beef-4a52-a98f-47414f15efe1-host-slash\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-dbus\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451165    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451203    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451242    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-k8s-cni-cncf-io\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451271    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-multus-certs\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451290    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-kubelet-config\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451304    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-hostroot\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451320    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk5rz\" (UniqueName: \"kubernetes.io/projected/01973b42-4064-4c7c-b6ef-b85d89b2f152-kube-api-access-rk5rz\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451340    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-kubernetes\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-run\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451402    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-socket-dir-parent\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451472    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-conf-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.451632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-device-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.452237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451521    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-var-lib-kubelet\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.452237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-tmp\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.452237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451563    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-slash\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.452237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451584    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-netns\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.452237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451605    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2d2d439-0d2f-4377-a043-f307916402f7-ovn-node-metrics-cert\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.452237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451629    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6z9\" (UniqueName: \"kubernetes.io/projected/f2d2d439-0d2f-4377-a043-f307916402f7-kube-api-access-qc6z9\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.452237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-conf\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.452237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451705    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5wf7\" (UniqueName: \"kubernetes.io/projected/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-kube-api-access-j5wf7\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:24.452237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.451739    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.475776 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.475732    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal" event={"ID":"72498d2be80a2b536a0b73a7eab91abc","Type":"ContainerStarted","Data":"041e79e9268d06e9f6c0ea23703be9d2adfc7838fe0f0332b090d7d15734ad12"}
Oct 30 02:40:24.497629 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.497494    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:40:24.497629 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.497564    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:40:24.497629 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.497563    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:40:24.497629 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.497623    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:40:24.498793 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.498384    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j5lkg\""
Oct 30 02:40:24.498793 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.498514    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:40:24.498793 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.498557    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:40:24.498793 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.498624    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-86l9w\""
Oct 30 02:40:24.498793 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.498640    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:24.498793 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.498739    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:40:24.498793 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.498748    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-prmc8\""
Oct 30 02:40:24.498793 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.498648    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:40:24.511277 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.511259    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:40:24.511387 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.511310    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:40:24.511387 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.511313    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:40:24.511527 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.511416    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:24.512178 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.512146    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:40:24.512263 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.512205    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r7gc2\""
Oct 30 02:40:24.543959 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.543940    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:24.552203 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552179    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpg5c\" (UniqueName: \"kubernetes.io/projected/15a18601-beef-4a52-a98f-47414f15efe1-kube-api-access-cpg5c\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.552305 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552210    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15a18601-beef-4a52-a98f-47414f15efe1-host-slash\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.552305 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552229    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-dbus\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:24.552305 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552252    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.552305 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552277    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.552305 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552299    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-k8s-cni-cncf-io\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.552540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-multus-certs\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.552540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552341    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-kubelet-config\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:24.552540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552367    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-hostroot\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.552540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552389    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5rz\" (UniqueName: \"kubernetes.io/projected/01973b42-4064-4c7c-b6ef-b85d89b2f152-kube-api-access-rk5rz\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.552540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552411    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-kubernetes\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.552540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552446    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-run\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.552540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552474    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-socket-dir-parent\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.552540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552495    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-conf-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.552540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552519    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-device-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.552540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552542    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-var-lib-kubelet\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.552898 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552554    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15a18601-beef-4a52-a98f-47414f15efe1-host-slash\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.552898 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-tmp\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.552898 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552790    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-dbus\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:24.552898 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552828    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-slash\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.552898 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552856    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-netns\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.553078 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552908    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2d2d439-0d2f-4377-a043-f307916402f7-ovn-node-metrics-cert\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.553078 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552914    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:40:24.553078 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6z9\" (UniqueName: \"kubernetes.io/projected/f2d2d439-0d2f-4377-a043-f307916402f7-kube-api-access-qc6z9\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.553078 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552957    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-conf\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.553078 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552977    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5wf7\" (UniqueName: \"kubernetes.io/projected/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-kube-api-access-j5wf7\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:24.553078 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.552998    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.553078 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-kubelet\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.553078 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553040    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6g6\" (UniqueName: \"kubernetes.io/projected/702224cc-b73b-468d-82e4-1647167c6314-kube-api-access-wg6g6\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.553078 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553059    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-netd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7schg\" (UniqueName: \"kubernetes.io/projected/54cc321a-a26d-4aab-8e95-144a2f9aaa41-kube-api-access-7schg\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553114    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-sys\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncl2\" (UniqueName: \"kubernetes.io/projected/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-kube-api-access-gncl2\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553174    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cnibin\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553198    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553224    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-kubelet\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553251    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-log-socket\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553274    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-env-overrides\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553303    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553331    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm7m6\" (UniqueName: \"kubernetes.io/projected/5f82d4e3-4636-4d13-9770-ae32b7719a7b-kube-api-access-tm7m6\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553356    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-cnibin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.553376 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553375    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553416    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01973b42-4064-4c7c-b6ef-b85d89b2f152-hosts-file\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553447    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-etc-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553469    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553492    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-systemd\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553506    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-system-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553520    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-cni-binary-copy\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553536    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553550    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553568    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-netns\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553597    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-systemd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553645    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-script-lib\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/729ae77b-bc29-4685-b33b-38feb7fe0a68-konnectivity-ca\") pod \"konnectivity-agent-crnh2\" (UID: \"729ae77b-bc29-4685-b33b-38feb7fe0a68\") " pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553698    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-host\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553714    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-modprobe-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553738    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-multus-daemon-config\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553763    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-etc-kubernetes\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.553805 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553778    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01973b42-4064-4c7c-b6ef-b85d89b2f152-tmp-dir\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553792    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-node-log\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-socket-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553876    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-kubelet-config\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553920    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/729ae77b-bc29-4685-b33b-38feb7fe0a68-agent-certs\") pod \"konnectivity-agent-crnh2\" (UID: \"729ae77b-bc29-4685-b33b-38feb7fe0a68\") " pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553931    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-socket-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553954    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysconfig\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553983    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-lib-modules\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.553988    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-bin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554038    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-systemd-units\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554067    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-var-lib-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554094    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-sys-fs\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554123    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54cc321a-a26d-4aab-8e95-144a2f9aaa41-host\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554150    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554179    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-multus\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554205    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-ovn\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.554408 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554231    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554286    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-config\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554341    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-registration-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554370    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldfr\" (UniqueName: \"kubernetes.io/projected/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kube-api-access-bldfr\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554396    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-tuned\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554423    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-system-cni-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554451    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54cc321a-a26d-4aab-8e95-144a2f9aaa41-serviceca\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554528    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-k8s-cni-cncf-io\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554533    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15a18601-beef-4a52-a98f-47414f15efe1-iptables-alerter-script\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554567    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-cnibin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554578    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-slash\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554576    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-netns\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554614    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554663    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.554993 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554662    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-run\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554701    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-kubernetes\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554762    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-multus-certs\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554783    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-systemd\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.554824    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-system-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555024    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01973b42-4064-4c7c-b6ef-b85d89b2f152-tmp-dir\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555096    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-device-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555110    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555145    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-var-lib-kubelet\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555156    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-etc-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555196    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-kubelet\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555232    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-cni-binary-copy\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555274    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-netd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.555566 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555473    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-sys\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.556072 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555639    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cnibin\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.556072 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555752    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.556072 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555782    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-kubelet\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.556072 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555781    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.556072 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555812    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-log-socket\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.556072 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555825    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-netns\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.556072 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.555856    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysconfig\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.556072 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.555896    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:24.556394 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556311    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-env-overrides\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.556394 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556348    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.556494 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.556400    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:24.556494 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556402    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-multus\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.556494 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556454    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-script-lib\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.556494 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556464    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-ovn\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.556494 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.556490    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs podName:2bc38ad4-6aa2-4d8d-a273-d1cae373266c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:25.056469186 +0000 UTC m=+3.076788416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs") pod "network-metrics-daemon-d9gmb" (UID: "2bc38ad4-6aa2-4d8d-a273-d1cae373266c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:24.556694 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556507    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-systemd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.556694 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556509    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.556694 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556542    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.556694 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556668    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01973b42-4064-4c7c-b6ef-b85d89b2f152-hosts-file\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.556871 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-bin\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.556871 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556743    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.556871 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556831    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-socket-dir-parent\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.557017 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.556989    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-host\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.557075 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557044    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-modprobe-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.557130 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557084    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-etc-kubernetes\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.557130 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557082    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-registration-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.557242 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557192    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-conf\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.557291 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557237    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-conf-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.557349 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557322    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-var-lib-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.557526 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557415    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-lib-modules\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.557526 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557471    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-bin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.557526 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557507    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-systemd-units\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.557526 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557506    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-sys-fs\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.557817 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557555    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54cc321a-a26d-4aab-8e95-144a2f9aaa41-host\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.557817 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.557558    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret podName:8308710a-3726-4a5e-90b5-7a6d63a1a25b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:25.057541774 +0000 UTC m=+3.077861002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret") pod "global-pull-secret-syncer-wttwd" (UID: "8308710a-3726-4a5e-90b5-7a6d63a1a25b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:24.557817 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557593    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-node-log\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.557817 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557592    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-system-cni-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.557817 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557603    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-hostroot\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.557817 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557710    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-bin\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.557817 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.557787    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-tmp\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.558172 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.558110    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/729ae77b-bc29-4685-b33b-38feb7fe0a68-agent-certs\") pod \"konnectivity-agent-crnh2\" (UID: \"729ae77b-bc29-4685-b33b-38feb7fe0a68\") " pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:24.558776 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.558757    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-config\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.558863 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.558813    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15a18601-beef-4a52-a98f-47414f15efe1-iptables-alerter-script\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.559176 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.559155    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/729ae77b-bc29-4685-b33b-38feb7fe0a68-konnectivity-ca\") pod \"konnectivity-agent-crnh2\" (UID: \"729ae77b-bc29-4685-b33b-38feb7fe0a68\") " pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:24.559247 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.559157    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54cc321a-a26d-4aab-8e95-144a2f9aaa41-serviceca\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.559328 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.559303    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-multus-daemon-config\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.560706 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.560684    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-tuned\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.561203 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.561181    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2d2d439-0d2f-4377-a043-f307916402f7-ovn-node-metrics-cert\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.573695 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.573644    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:24.573695 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.573671    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:24.573695 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.573684    2571 projected.go:196] Error preparing data for projected volume kube-api-access-db4lq for pod openshift-network-diagnostics/network-check-target-k684d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:24.573903 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:24.573735    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq podName:eb12906b-59f7-473c-ad0b-c87929d603e4 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:25.073719657 +0000 UTC m=+3.094038885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-db4lq" (UniqueName: "kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq") pod "network-check-target-k684d" (UID: "eb12906b-59f7-473c-ad0b-c87929d603e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:24.575724 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.575539    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm7m6\" (UniqueName: \"kubernetes.io/projected/5f82d4e3-4636-4d13-9770-ae32b7719a7b-kube-api-access-tm7m6\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.575850 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.575705    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7schg\" (UniqueName: \"kubernetes.io/projected/54cc321a-a26d-4aab-8e95-144a2f9aaa41-kube-api-access-7schg\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.576059 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.576039    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncl2\" (UniqueName: \"kubernetes.io/projected/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-kube-api-access-gncl2\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.577356 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.577323    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpg5c\" (UniqueName: \"kubernetes.io/projected/15a18601-beef-4a52-a98f-47414f15efe1-kube-api-access-cpg5c\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.577481 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.577462    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6g6\" (UniqueName: \"kubernetes.io/projected/702224cc-b73b-468d-82e4-1647167c6314-kube-api-access-wg6g6\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.580054 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.580030    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5wf7\" (UniqueName: \"kubernetes.io/projected/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-kube-api-access-j5wf7\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:24.580597 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.580565    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldfr\" (UniqueName: \"kubernetes.io/projected/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kube-api-access-bldfr\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.581043 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.581019    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk5rz\" (UniqueName: \"kubernetes.io/projected/01973b42-4064-4c7c-b6ef-b85d89b2f152-kube-api-access-rk5rz\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.581179 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.581161    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6z9\" (UniqueName: \"kubernetes.io/projected/f2d2d439-0d2f-4377-a043-f307916402f7-kube-api-access-qc6z9\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.685473 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.685437    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:40:24.694368 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:24.694339    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fe1d12_9e94_4e41_8490_dc7f65b5a1a6.slice/crio-0cef657654452e5d47cebbecfd30a6e82eb56790a205f35ca6748ccdba7edffc WatchSource:0}: Error finding container 0cef657654452e5d47cebbecfd30a6e82eb56790a205f35ca6748ccdba7edffc: Status 404 returned error can't find the container with id 0cef657654452e5d47cebbecfd30a6e82eb56790a205f35ca6748ccdba7edffc
Oct 30 02:40:24.701297 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.701276    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:40:24.720340 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.720317    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:24.720449 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.720408    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nlhw8"
Oct 30 02:40:24.722388 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.722370    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:40:24.728192 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.728170    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:40:24.736720 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.736628    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:40:24.737476 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:24.737451    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ae74bc_1468_4326_b1bc_c6e640b5abd4.slice/crio-1325e70da97fd8655b5e6bb11036c1f1d8c301b583aac316d9cc5a9abeef24fb WatchSource:0}: Error finding container 1325e70da97fd8655b5e6bb11036c1f1d8c301b583aac316d9cc5a9abeef24fb: Status 404 returned error can't find the container with id 1325e70da97fd8655b5e6bb11036c1f1d8c301b583aac316d9cc5a9abeef24fb
Oct 30 02:40:24.745058 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:24.745027    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729ae77b_bc29_4685_b33b_38feb7fe0a68.slice/crio-2cbeac0618c785897f48a1544b9e440850a670db635b8099fc23d239f50c0fa8 WatchSource:0}: Error finding container 2cbeac0618c785897f48a1544b9e440850a670db635b8099fc23d239f50c0fa8: Status 404 returned error can't find the container with id 2cbeac0618c785897f48a1544b9e440850a670db635b8099fc23d239f50c0fa8
Oct 30 02:40:24.749573 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:24.749463    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01973b42_4064_4c7c_b6ef_b85d89b2f152.slice/crio-86187fbbdeccd22c99dbcef68042190fae77aa6858f61592ba702cb20bc0d859 WatchSource:0}: Error finding container 86187fbbdeccd22c99dbcef68042190fae77aa6858f61592ba702cb20bc0d859: Status 404 returned error can't find the container with id 86187fbbdeccd22c99dbcef68042190fae77aa6858f61592ba702cb20bc0d859
Oct 30 02:40:24.749956 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:24.749935    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f82d4e3_4636_4d13_9770_ae32b7719a7b.slice/crio-85af4cb15541f4a7d77c6922e5ee34d913889cab64a8c1aced97741eafc00fb1 WatchSource:0}: Error finding container 85af4cb15541f4a7d77c6922e5ee34d913889cab64a8c1aced97741eafc00fb1: Status 404 returned error can't find the container with id 85af4cb15541f4a7d77c6922e5ee34d913889cab64a8c1aced97741eafc00fb1
Oct 30 02:40:24.753600 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:24.753570    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cc321a_a26d_4aab_8e95_144a2f9aaa41.slice/crio-65af3ee82b74fbe246fd1f0d8ec2c18839506aba7f7b90af65e1524ba24fe5ee WatchSource:0}: Error finding container 65af3ee82b74fbe246fd1f0d8ec2c18839506aba7f7b90af65e1524ba24fe5ee: Status 404 returned error can't find the container with id 65af3ee82b74fbe246fd1f0d8ec2c18839506aba7f7b90af65e1524ba24fe5ee
Oct 30 02:40:24.755114 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.754954    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:40:24.759337 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:24.759315    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:24.769361 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:24.769337    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a18601_beef_4a52_a98f_47414f15efe1.slice/crio-954fbbbee2472bb1d4ae83ca2b4de19d60c2f61c2783168d033adaa421159bd9 WatchSource:0}: Error finding container 954fbbbee2472bb1d4ae83ca2b4de19d60c2f61c2783168d033adaa421159bd9: Status 404 returned error can't find the container with id 954fbbbee2472bb1d4ae83ca2b4de19d60c2f61c2783168d033adaa421159bd9
Oct 30 02:40:24.772568 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:24.772545    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d2d439_0d2f_4377_a043_f307916402f7.slice/crio-1d3c604880f128c7d323fb77a2981a67dd60a4a757e220a4a5d8b336d6f3194d WatchSource:0}: Error finding container 1d3c604880f128c7d323fb77a2981a67dd60a4a757e220a4a5d8b336d6f3194d: Status 404 returned error can't find the container with id 1d3c604880f128c7d323fb77a2981a67dd60a4a757e220a4a5d8b336d6f3194d
Oct 30 02:40:25.060684 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.060404    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:25.060841 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:25.060560    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:25.060841 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.060738    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:25.060841 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:25.060804    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs podName:2bc38ad4-6aa2-4d8d-a273-d1cae373266c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:26.060784617 +0000 UTC m=+4.081103851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs") pod "network-metrics-daemon-d9gmb" (UID: "2bc38ad4-6aa2-4d8d-a273-d1cae373266c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:25.061026 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:25.060856    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:25.061026 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:25.060899    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret podName:8308710a-3726-4a5e-90b5-7a6d63a1a25b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:26.060887198 +0000 UTC m=+4.081206426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret") pod "global-pull-secret-syncer-wttwd" (UID: "8308710a-3726-4a5e-90b5-7a6d63a1a25b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:25.100733 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.100704    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:25.162153 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.161609    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:25.162153 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:25.161738    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:25.162153 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:25.161751    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:25.162153 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:25.161763    2571 projected.go:196] Error preparing data for projected volume kube-api-access-db4lq for pod openshift-network-diagnostics/network-check-target-k684d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:25.162153 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:25.161816    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq podName:eb12906b-59f7-473c-ad0b-c87929d603e4 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:26.161803365 +0000 UTC m=+4.182122592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-db4lq" (UniqueName: "kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq") pod "network-check-target-k684d" (UID: "eb12906b-59f7-473c-ad0b-c87929d603e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:25.428663 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.428553    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:24 +0000 UTC" deadline="2028-05-20 18:08:12.23891383 +0000 UTC"
Oct 30 02:40:25.428663 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.428610    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13647h27m46.810307985s"
Oct 30 02:40:25.503880 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.503768    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ds5x7" event={"ID":"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6","Type":"ContainerStarted","Data":"0cef657654452e5d47cebbecfd30a6e82eb56790a205f35ca6748ccdba7edffc"}
Oct 30 02:40:25.517399 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.516537    2571 generic.go:356] "Generic (PLEG): container finished" podID="72498d2be80a2b536a0b73a7eab91abc" containerID="85a7cc084fbd7e6eb17b5886a8a8ca1cccbe1ce920b5cfcc0be86db02849f967" exitCode=0
Oct 30 02:40:25.517399 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.516620    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal" event={"ID":"72498d2be80a2b536a0b73a7eab91abc","Type":"ContainerDied","Data":"85a7cc084fbd7e6eb17b5886a8a8ca1cccbe1ce920b5cfcc0be86db02849f967"}
Oct 30 02:40:25.538554 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.538477    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-28mns" event={"ID":"54cc321a-a26d-4aab-8e95-144a2f9aaa41","Type":"ContainerStarted","Data":"65af3ee82b74fbe246fd1f0d8ec2c18839506aba7f7b90af65e1524ba24fe5ee"}
Oct 30 02:40:25.554405 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.554374    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snstr" event={"ID":"5f82d4e3-4636-4d13-9770-ae32b7719a7b","Type":"ContainerStarted","Data":"85af4cb15541f4a7d77c6922e5ee34d913889cab64a8c1aced97741eafc00fb1"}
Oct 30 02:40:25.566834 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.566799    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-76z2q" event={"ID":"01973b42-4064-4c7c-b6ef-b85d89b2f152","Type":"ContainerStarted","Data":"86187fbbdeccd22c99dbcef68042190fae77aa6858f61592ba702cb20bc0d859"}
Oct 30 02:40:25.581646 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.581611    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr" event={"ID":"50ae74bc-1468-4326-b1bc-c6e640b5abd4","Type":"ContainerStarted","Data":"1325e70da97fd8655b5e6bb11036c1f1d8c301b583aac316d9cc5a9abeef24fb"}
Oct 30 02:40:25.596062 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.595367    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal" event={"ID":"4b5a364f76591020026198c5396db417","Type":"ContainerStarted","Data":"cb5cac9fa8d20ea32d27f6c8b20978009aa4053c9feb5905b86f0d70ee0ed80f"}
Oct 30 02:40:25.602693 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.602590    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j" event={"ID":"f2d2d439-0d2f-4377-a043-f307916402f7","Type":"ContainerStarted","Data":"1d3c604880f128c7d323fb77a2981a67dd60a4a757e220a4a5d8b336d6f3194d"}
Oct 30 02:40:25.605219 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.605142    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tdp5t" event={"ID":"15a18601-beef-4a52-a98f-47414f15efe1","Type":"ContainerStarted","Data":"954fbbbee2472bb1d4ae83ca2b4de19d60c2f61c2783168d033adaa421159bd9"}
Oct 30 02:40:25.632092 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.631995    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-crnh2" event={"ID":"729ae77b-bc29-4685-b33b-38feb7fe0a68","Type":"ContainerStarted","Data":"2cbeac0618c785897f48a1544b9e440850a670db635b8099fc23d239f50c0fa8"}
Oct 30 02:40:25.647711 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:25.647638    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nlhw8" event={"ID":"702224cc-b73b-468d-82e4-1647167c6314","Type":"ContainerStarted","Data":"be3a657b3f6bec0b57c5185354f0e4af4364edd118182482532eb6bdd54cdee0"}
Oct 30 02:40:26.068706 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:26.068669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:26.068901 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:26.068734    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:26.068901 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.068868    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:26.069016 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.068925    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret podName:8308710a-3726-4a5e-90b5-7a6d63a1a25b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:28.068908329 +0000 UTC m=+6.089227546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret") pod "global-pull-secret-syncer-wttwd" (UID: "8308710a-3726-4a5e-90b5-7a6d63a1a25b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:26.069313 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.069293    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:26.069405 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.069350    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs podName:2bc38ad4-6aa2-4d8d-a273-d1cae373266c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:28.069333576 +0000 UTC m=+6.089652792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs") pod "network-metrics-daemon-d9gmb" (UID: "2bc38ad4-6aa2-4d8d-a273-d1cae373266c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:26.170097 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:26.170005    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:26.170255 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.170170    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:26.170255 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.170194    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:26.170255 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.170206    2571 projected.go:196] Error preparing data for projected volume kube-api-access-db4lq for pod openshift-network-diagnostics/network-check-target-k684d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:26.170443 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.170258    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq podName:eb12906b-59f7-473c-ad0b-c87929d603e4 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:28.170240775 +0000 UTC m=+6.190559991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-db4lq" (UniqueName: "kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq") pod "network-check-target-k684d" (UID: "eb12906b-59f7-473c-ad0b-c87929d603e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:26.429239 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:26.429129    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:24 +0000 UTC" deadline="2028-06-26 03:14:21.597755909 +0000 UTC"
Oct 30 02:40:26.429239 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:26.429196    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14520h33m55.168564349s"
Oct 30 02:40:26.474089 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:26.474045    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:26.474271 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.474198    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:26.474632 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:26.474612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:26.474737 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.474716    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:26.474800 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:26.474783    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:26.474882 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:26.474850    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:26.662037 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:26.661999    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal" event={"ID":"72498d2be80a2b536a0b73a7eab91abc","Type":"ContainerStarted","Data":"9e7a62ac7609a3498b1d64d28af7dee52dd38731c248adbcbcebc334f097f86c"}
Oct 30 02:40:26.702368 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:26.702265    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal" podStartSLOduration=3.702244336 podStartE2EDuration="3.702244336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:25.642019257 +0000 UTC m=+3.662338497" watchObservedRunningTime="2026-10-30 02:40:26.702244336 +0000 UTC m=+4.722563574"
Oct 30 02:40:27.672020 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:27.671121    2571 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="0ebed9e7fa5622697d7bb29a2394adc51f178ea6c66c8c74d251659fb211cc9b" exitCode=0
Oct 30 02:40:27.672020 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:27.671530    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snstr" event={"ID":"5f82d4e3-4636-4d13-9770-ae32b7719a7b","Type":"ContainerDied","Data":"0ebed9e7fa5622697d7bb29a2394adc51f178ea6c66c8c74d251659fb211cc9b"}
Oct 30 02:40:27.745308 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:27.745190    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal" podStartSLOduration=4.745167953 podStartE2EDuration="4.745167953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:26.702807535 +0000 UTC m=+4.723126762" watchObservedRunningTime="2026-10-30 02:40:27.745167953 +0000 UTC m=+5.765487190"
Oct 30 02:40:28.086600 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:28.086445    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:28.086600 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:28.086526    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:28.086600 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.086590    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:28.086887 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.086660    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs podName:2bc38ad4-6aa2-4d8d-a273-d1cae373266c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.086642293 +0000 UTC m=+10.106961517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs") pod "network-metrics-daemon-d9gmb" (UID: "2bc38ad4-6aa2-4d8d-a273-d1cae373266c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:28.086887 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.086679    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:28.086887 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.086772    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret podName:8308710a-3726-4a5e-90b5-7a6d63a1a25b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.086727391 +0000 UTC m=+10.107046624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret") pod "global-pull-secret-syncer-wttwd" (UID: "8308710a-3726-4a5e-90b5-7a6d63a1a25b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:28.188367 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:28.187706    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:28.188367 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.187911    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:28.188367 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.187936    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:28.188367 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.187949    2571 projected.go:196] Error preparing data for projected volume kube-api-access-db4lq for pod openshift-network-diagnostics/network-check-target-k684d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:28.188367 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.188010    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq podName:eb12906b-59f7-473c-ad0b-c87929d603e4 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.187992394 +0000 UTC m=+10.208311622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-db4lq" (UniqueName: "kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq") pod "network-check-target-k684d" (UID: "eb12906b-59f7-473c-ad0b-c87929d603e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:28.473347 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:28.473318    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:28.473520 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.473470    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:28.473873 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:28.473821    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:28.473974 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.473913    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:28.473974 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:28.473958    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:28.474077 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:28.474023    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:28.679463 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:28.679355    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-crnh2" event={"ID":"729ae77b-bc29-4685-b33b-38feb7fe0a68","Type":"ContainerStarted","Data":"4de9869356f1a821e1757d3ba669a61776063528e3379bc18e598578434d7e65"}
Oct 30 02:40:30.473644 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:30.473573    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:30.473644 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:30.473610    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:30.474138 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:30.473573    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:30.474138 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:30.473687    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:30.474138 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:30.473811    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:30.474138 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:30.473898    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:30.686396 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:30.686313    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-28mns" event={"ID":"54cc321a-a26d-4aab-8e95-144a2f9aaa41","Type":"ContainerStarted","Data":"4c5e91e79a538a7ee5f233a06d8823e1d3215fb2a00ba8c481eabacc94465d58"}
Oct 30 02:40:30.696679 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:30.696648    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr" event={"ID":"50ae74bc-1468-4326-b1bc-c6e640b5abd4","Type":"ContainerStarted","Data":"b992a7837f57a88184fe3635823989b6a2527a9b20f27b6c5d759a1b6bbcc6c7"}
Oct 30 02:40:30.727104 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:30.727051    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-crnh2" podStartSLOduration=5.161390147 podStartE2EDuration="8.727033236s" totalImagesPullingTime="3.565643089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:28.721244378 +0000 UTC m=+6.741563615" watchObservedRunningTime="2026-10-30 02:40:30.727033236 +0000 UTC m=+8.747352463"
Oct 30 02:40:31.594497 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:31.594460    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:31.595665 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:31.595644    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:31.636296 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:31.636238    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-28mns" podStartSLOduration=4.648258453 podStartE2EDuration="9.636219111s" totalImagesPullingTime="4.987960658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:30.728344662 +0000 UTC m=+8.748663898" watchObservedRunningTime="2026-10-30 02:40:31.636219111 +0000 UTC m=+9.656538350"
Oct 30 02:40:31.983493 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:31.983466    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:40:32.137317 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.137285    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:32.137444 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.137361    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:32.137521 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.137488    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:32.137572 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.137544    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs podName:2bc38ad4-6aa2-4d8d-a273-d1cae373266c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:40.137525956 +0000 UTC m=+18.157845187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs") pod "network-metrics-daemon-d9gmb" (UID: "2bc38ad4-6aa2-4d8d-a273-d1cae373266c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:32.137635 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.137604    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:32.137687 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.137636    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret podName:8308710a-3726-4a5e-90b5-7a6d63a1a25b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:40.137625268 +0000 UTC m=+18.157944498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret") pod "global-pull-secret-syncer-wttwd" (UID: "8308710a-3726-4a5e-90b5-7a6d63a1a25b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:32.238381 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.238350    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:32.238595 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.238565    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:32.238595 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.238590    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:32.238701 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.238603    2571 projected.go:196] Error preparing data for projected volume kube-api-access-db4lq for pod openshift-network-diagnostics/network-check-target-k684d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:32.238701 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.238667    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq podName:eb12906b-59f7-473c-ad0b-c87929d603e4 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:40.238648497 +0000 UTC m=+18.258967714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-db4lq" (UniqueName: "kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq") pod "network-check-target-k684d" (UID: "eb12906b-59f7-473c-ad0b-c87929d603e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:32.405008 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.404857    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:40:31.983487732Z","UUID":"28656061-6d4b-4dab-8994-c208d1b6d8b3","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:40:32.406922 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.406810    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:40:32.406922 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.406838    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:40:32.473051 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.471767    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:32.473051 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.471879    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:32.473051 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.472224    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:32.473051 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.472310    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:32.473051 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.472378    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:32.473051 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:32.472506    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:32.705871 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.705179    2571 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="deb09f6110c575fa5d1c0b6833652b7fc4a057f3c9ec2a82291b1752fda79299" exitCode=0
Oct 30 02:40:32.705871 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.705380    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snstr" event={"ID":"5f82d4e3-4636-4d13-9770-ae32b7719a7b","Type":"ContainerDied","Data":"deb09f6110c575fa5d1c0b6833652b7fc4a057f3c9ec2a82291b1752fda79299"}
Oct 30 02:40:32.709860 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.709835    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-76z2q" event={"ID":"01973b42-4064-4c7c-b6ef-b85d89b2f152","Type":"ContainerStarted","Data":"4defca215029b023ed7207cadcf05565d5c289ebe852aaf728492323bd0ad090"}
Oct 30 02:40:32.712478 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.712454    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr" event={"ID":"50ae74bc-1468-4326-b1bc-c6e640b5abd4","Type":"ContainerStarted","Data":"1b6ff1a503ce8a7de2450173ace30ebd271c83cdafa54ffe6287d78f4e589675"}
Oct 30 02:40:32.715274 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.715249    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nlhw8" event={"ID":"702224cc-b73b-468d-82e4-1647167c6314","Type":"ContainerStarted","Data":"060161d030e8b7ec85687be9ac01645bd0e39c80e98a6b4c711408a59f1f2b6b"}
Oct 30 02:40:32.834009 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.833952    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-76z2q" podStartSLOduration=3.168885358 podStartE2EDuration="10.833932971s" totalImagesPullingTime="7.665047613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:32.791998064 +0000 UTC m=+10.812317301" watchObservedRunningTime="2026-10-30 02:40:32.833932971 +0000 UTC m=+10.854252208"
Oct 30 02:40:32.834372 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:32.834338    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nlhw8" podStartSLOduration=3.314482984 podStartE2EDuration="10.834329163s" totalImagesPullingTime="7.519846179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:32.83352844 +0000 UTC m=+10.853847673" watchObservedRunningTime="2026-10-30 02:40:32.834329163 +0000 UTC m=+10.854648400"
Oct 30 02:40:33.726302 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:33.726213    2571 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="478d264a9262f2a2861032d5c0ec4562f18cd2649ebe5c4d6e54aa9092e5363b" exitCode=0
Oct 30 02:40:33.726720 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:33.726325    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snstr" event={"ID":"5f82d4e3-4636-4d13-9770-ae32b7719a7b","Type":"ContainerDied","Data":"478d264a9262f2a2861032d5c0ec4562f18cd2649ebe5c4d6e54aa9092e5363b"}
Oct 30 02:40:33.729690 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:33.729604    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr" event={"ID":"50ae74bc-1468-4326-b1bc-c6e640b5abd4","Type":"ContainerStarted","Data":"844c4c8bd2be7a365e48a24ba2537fded53cea3979fcbeccceebde053c1678ac"}
Oct 30 02:40:33.830801 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:33.830753    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr" podStartSLOduration=4.045101894 podStartE2EDuration="11.83073913s" totalImagesPullingTime="7.785637236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:33.830518653 +0000 UTC m=+11.850837890" watchObservedRunningTime="2026-10-30 02:40:33.83073913 +0000 UTC m=+11.851058366"
Oct 30 02:40:34.471383 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:34.471359    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:34.471500 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:34.471359    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:34.471563 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:34.471543    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:34.471952 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:34.471931    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:34.472012 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:34.471999    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:34.472093 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:34.472077    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:34.732897 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:34.732820    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ds5x7" event={"ID":"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6","Type":"ContainerStarted","Data":"1cd059e32221ab27d17e64a9f4629440efc6ecee3c63db7801c9162207ed0692"}
Oct 30 02:40:34.734508 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:34.734485    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tdp5t" event={"ID":"15a18601-beef-4a52-a98f-47414f15efe1","Type":"ContainerStarted","Data":"6fdd23c3dd3c190759fa940ee57db3b91a212513ad53cd5fdc52313e05079ee5"}
Oct 30 02:40:34.780626 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:34.780313    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ds5x7" podStartSLOduration=3.071301722 podStartE2EDuration="12.780296891s" totalImagesPullingTime="9.708995169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:34.77970889 +0000 UTC m=+12.800028127" watchObservedRunningTime="2026-10-30 02:40:34.780296891 +0000 UTC m=+12.800616124"
Oct 30 02:40:34.819965 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:34.819918    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tdp5t" podStartSLOduration=4.939918089 podStartE2EDuration="12.819901153s" totalImagesPullingTime="7.879983064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:34.81954108 +0000 UTC m=+12.839860316" watchObservedRunningTime="2026-10-30 02:40:34.819901153 +0000 UTC m=+12.840220390"
Oct 30 02:40:35.739970 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:35.739930    2571 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="d25a5a69e3b4e7aa45375553ab8b2a999925e4987094cf7b2331c9d7e141df94" exitCode=0
Oct 30 02:40:35.740583 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:35.740004    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snstr" event={"ID":"5f82d4e3-4636-4d13-9770-ae32b7719a7b","Type":"ContainerDied","Data":"d25a5a69e3b4e7aa45375553ab8b2a999925e4987094cf7b2331c9d7e141df94"}
Oct 30 02:40:36.471997 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:36.471960    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:36.472184 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:36.472112    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:36.472548 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:36.472528    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:36.472672 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:36.472615    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:36.472963 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:36.472940    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:36.473066 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:36.473021    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:37.747463 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:37.747411    2571 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="c697a308cd9ba26f1627e4f27c73a53f1ba187523a64c8c09ba442182b29da64" exitCode=0
Oct 30 02:40:37.747922 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:37.747469    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snstr" event={"ID":"5f82d4e3-4636-4d13-9770-ae32b7719a7b","Type":"ContainerDied","Data":"c697a308cd9ba26f1627e4f27c73a53f1ba187523a64c8c09ba442182b29da64"}
Oct 30 02:40:38.471392 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:38.471357    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:38.471392 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:38.471390    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:38.471694 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:38.471410    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:38.471694 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:38.471543    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:38.471694 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:38.471594    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:38.471834 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:38.471702    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:38.752311 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:38.752233    2571 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="3736a450088187de1cd822378b7b3b629a69d43264d639e9a9d5eca9405713a4" exitCode=0
Oct 30 02:40:38.752311 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:38.752285    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snstr" event={"ID":"5f82d4e3-4636-4d13-9770-ae32b7719a7b","Type":"ContainerDied","Data":"3736a450088187de1cd822378b7b3b629a69d43264d639e9a9d5eca9405713a4"}
Oct 30 02:40:39.757106 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:39.756928    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j" event={"ID":"f2d2d439-0d2f-4377-a043-f307916402f7","Type":"ContainerStarted","Data":"5794e05170779a0158c91b6b91763d18ee7c8ac886b631f02393f8e863361287"}
Oct 30 02:40:39.757547 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:39.757154    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j" event={"ID":"f2d2d439-0d2f-4377-a043-f307916402f7","Type":"ContainerStarted","Data":"0c269c6c7e19b387a974ef580d62733c5cbaaf2a3f6d899af9cdd86eba0c7a9a"}
Oct 30 02:40:39.757547 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:39.757188    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j" event={"ID":"f2d2d439-0d2f-4377-a043-f307916402f7","Type":"ContainerStarted","Data":"6684308b2ef64810d9b551bf842dada315c8ffd5c416cf0c0a76203c6cd6d22f"}
Oct 30 02:40:39.757547 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:39.757221    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j" event={"ID":"f2d2d439-0d2f-4377-a043-f307916402f7","Type":"ContainerStarted","Data":"e7f67556a8b842bc19c4739a29a948941c55497b938472e2ff628d4c6ce33100"}
Oct 30 02:40:39.757547 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:39.757248    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j" event={"ID":"f2d2d439-0d2f-4377-a043-f307916402f7","Type":"ContainerStarted","Data":"6509ec3174ec0045e5493cebe6c918a93f45dffdabc4a9fa67b2db4d6ba6a1aa"}
Oct 30 02:40:39.757547 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:39.757282    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j" event={"ID":"f2d2d439-0d2f-4377-a043-f307916402f7","Type":"ContainerStarted","Data":"993439b1dc0b8982bc5dd18b4d31ffd8add8feee0824d704125bcc6d26d4285b"}
Oct 30 02:40:39.760221 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:39.760199    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snstr" event={"ID":"5f82d4e3-4636-4d13-9770-ae32b7719a7b","Type":"ContainerStarted","Data":"14e7cb83b8213897798f979f8ed2c01e88c37fc8f8690fd5e6955d31f3fc9db5"}
Oct 30 02:40:39.806293 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:39.806256    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-snstr" podStartSLOduration=7.421889537 podStartE2EDuration="17.806245265s" totalImagesPullingTime="10.384355728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:39.805834548 +0000 UTC m=+17.826153783" watchObservedRunningTime="2026-10-30 02:40:39.806245265 +0000 UTC m=+17.826564501"
Oct 30 02:40:40.198916 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:40.198814    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:40.201558 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:40.199225    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:40.201558 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.199314    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:40.201558 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.199388    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:40.201558 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.199402    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs podName:2bc38ad4-6aa2-4d8d-a273-d1cae373266c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:56.199382078 +0000 UTC m=+34.219701307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs") pod "network-metrics-daemon-d9gmb" (UID: "2bc38ad4-6aa2-4d8d-a273-d1cae373266c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:40.201558 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.199473    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret podName:8308710a-3726-4a5e-90b5-7a6d63a1a25b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:56.199457627 +0000 UTC m=+34.219776858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret") pod "global-pull-secret-syncer-wttwd" (UID: "8308710a-3726-4a5e-90b5-7a6d63a1a25b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:40.300126 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:40.300096    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:40.300270 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.300226    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:40.300270 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.300240    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:40.300270 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.300249    2571 projected.go:196] Error preparing data for projected volume kube-api-access-db4lq for pod openshift-network-diagnostics/network-check-target-k684d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:40.300375 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.300295    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq podName:eb12906b-59f7-473c-ad0b-c87929d603e4 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:56.300283524 +0000 UTC m=+34.320602752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-db4lq" (UniqueName: "kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq") pod "network-check-target-k684d" (UID: "eb12906b-59f7-473c-ad0b-c87929d603e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:40.471222 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:40.471147    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:40.471371 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:40.471154    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:40.471371 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.471261    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:40.471371 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:40.471283    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:40.471371 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.471364    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:40.471582 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:40.471419    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:41.767637 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:41.767603    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j" event={"ID":"f2d2d439-0d2f-4377-a043-f307916402f7","Type":"ContainerStarted","Data":"9973496544f1be56b346514af6a36e39814060f933b1b3aa0a52ee12b325cf54"}
Oct 30 02:40:42.472261 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:42.472225    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:42.472419 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:42.472308    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:42.472419 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:42.472386    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:42.472611 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:42.472476    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:42.472611 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:42.472511    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:42.472611 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:42.472563    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:43.775804 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:43.775774    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j" event={"ID":"f2d2d439-0d2f-4377-a043-f307916402f7","Type":"ContainerStarted","Data":"04c19d549e01c93f5ec53669e4dc225573ffd3ea68f7268c0f0df16c7ae5b01c"}
Oct 30 02:40:43.776451 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:43.776032    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:43.791570 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:43.791544    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:43.873989 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:43.873883    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j" podStartSLOduration=7.459075408 podStartE2EDuration="21.873870812s" totalImagesPullingTime="14.414795404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:43.873469598 +0000 UTC m=+21.893788834" watchObservedRunningTime="2026-10-30 02:40:43.873870812 +0000 UTC m=+21.894190048"
Oct 30 02:40:44.471540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:44.471509    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:44.471540 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:44.471528    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:44.471776 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:44.471637    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:44.471776 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:44.471724    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:44.471846 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:44.471816    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:44.471914 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:44.471897    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:44.777765 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:44.777736    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:44.778221 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:44.777803    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:44.792235 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:44.792213    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:40:45.705577 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:45.705503    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wttwd"]
Oct 30 02:40:45.705929 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:45.705910    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:45.706168 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:45.706146    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:45.709989 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:45.709955    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d9gmb"]
Oct 30 02:40:45.710085 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:45.710068    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:45.710176 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:45.710158    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:45.710234 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:45.710190    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k684d"]
Oct 30 02:40:45.710301 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:45.710288    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:45.710409 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:45.710390    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:47.471684 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:47.471653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:47.472237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:47.471653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:47.472237 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:47.471761    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:47.472237 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:47.471858    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:47.472237 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:47.471662    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:47.472237 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:47.471954    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:49.471604 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:49.471569    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:49.471604 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:49.471598    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:49.472072 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:49.471573    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:49.472072 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:49.471663    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:49.472072 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:49.471770    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:49.472072 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:49.471842    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:51.471213 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:51.471173    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:51.471213 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:51.471198    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:51.471709 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:51.471279    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:51.471709 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:51.471285    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k684d" podUID="eb12906b-59f7-473c-ad0b-c87929d603e4"
Oct 30 02:40:51.471709 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:51.471340    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wttwd" podUID="8308710a-3726-4a5e-90b5-7a6d63a1a25b"
Oct 30 02:40:51.471709 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:51.471462    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9gmb" podUID="2bc38ad4-6aa2-4d8d-a273-d1cae373266c"
Oct 30 02:40:51.766374 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:51.766351    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeReady"
Oct 30 02:40:51.766513 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:51.766486    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 02:40:52.697816 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:52.697789    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-76z2q_01973b42-4064-4c7c-b6ef-b85d89b2f152/dns-node-resolver/0.log"
Oct 30 02:40:53.470787 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.470756    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:53.470975 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.470757    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:53.471139 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.470757    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:53.487854 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.487837    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:40:53.487854 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.487845    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:40:53.487965 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.487873    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qc2sc\""
Oct 30 02:40:53.502001 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.501982    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:40:53.502175 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.502159    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:40:53.502227 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.502199    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kk2sv\""
Oct 30 02:40:53.502227 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.502214    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xsmz\""
Oct 30 02:40:53.646862 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:53.646839    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-28mns_54cc321a-a26d-4aab-8e95-144a2f9aaa41/node-ca/0.log"
Oct 30 02:40:54.685006 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:54.684972    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:54.685495 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:54.685093    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:40:54.685609 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:54.685593    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:40:56.213623 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:56.213585    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:56.214045 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:56.213645    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:40:56.214045 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:56.213736    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:40:56.214045 ip-10-0-128-214 kubenswrapper[2571]: E1030 02:40:56.213790    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs podName:2bc38ad4-6aa2-4d8d-a273-d1cae373266c nodeName:}" failed. No retries permitted until 2026-10-30 02:41:28.213776705 +0000 UTC m=+66.234095931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs") pod "network-metrics-daemon-d9gmb" (UID: "2bc38ad4-6aa2-4d8d-a273-d1cae373266c") : secret "metrics-daemon-secret" not found
Oct 30 02:40:56.216898 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:56.216872    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:56.314054 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:56.314019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:56.316606 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:56.316576    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:56.513144 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:56.513122    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:40:56.514210 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:56.514187    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:40:56.963731 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:56.963648    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wttwd"]
Oct 30 02:40:56.967649 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:56.967615    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8308710a_3726_4a5e_90b5_7a6d63a1a25b.slice/crio-1e2b71f3d259eda648f648548cdd288d42be7e2fcf0c39ba96acd4b5908c82f1 WatchSource:0}: Error finding container 1e2b71f3d259eda648f648548cdd288d42be7e2fcf0c39ba96acd4b5908c82f1: Status 404 returned error can't find the container with id 1e2b71f3d259eda648f648548cdd288d42be7e2fcf0c39ba96acd4b5908c82f1
Oct 30 02:40:56.973418 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:56.973394    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k684d"]
Oct 30 02:40:56.975301 ip-10-0-128-214 kubenswrapper[2571]: W1030 02:40:56.975278    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb12906b_59f7_473c_ad0b_c87929d603e4.slice/crio-b63509df8040379b61d43121cbad4b87d4f970d307681aba00cbc384f6d349f0 WatchSource:0}: Error finding container b63509df8040379b61d43121cbad4b87d4f970d307681aba00cbc384f6d349f0: Status 404 returned error can't find the container with id b63509df8040379b61d43121cbad4b87d4f970d307681aba00cbc384f6d349f0
Oct 30 02:40:57.805593 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:57.805551    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k684d" event={"ID":"eb12906b-59f7-473c-ad0b-c87929d603e4","Type":"ContainerStarted","Data":"b63509df8040379b61d43121cbad4b87d4f970d307681aba00cbc384f6d349f0"}
Oct 30 02:40:57.806911 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:40:57.806881    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wttwd" event={"ID":"8308710a-3726-4a5e-90b5-7a6d63a1a25b","Type":"ContainerStarted","Data":"1e2b71f3d259eda648f648548cdd288d42be7e2fcf0c39ba96acd4b5908c82f1"}
Oct 30 02:41:00.817317 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:41:00.817270    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k684d" event={"ID":"eb12906b-59f7-473c-ad0b-c87929d603e4","Type":"ContainerStarted","Data":"1fbb90b1f3a45205dbfcbc884e2d603ac6d01f735d6ee5bb9635658f524b2180"}
Oct 30 02:41:00.817783 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:41:00.817461    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:41:00.858206 ip-10-0-128-214 kubenswrapper[2571]: I1030 02:41:00.858159    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k684d" podStartSLOduration=35.4671424 podStartE2EDuration="38.858143544s" totalImagesPullingTime="3.391001144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:00.856955398 +0000 UTC m=+38.877274644" watchObservedRunningTime="2026-10-30 02:41:00.858143544 +0000 UTC m=+38.878462779"
Oct 30 02:41:02.009520 ip-10-0-128-214 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 02:41:02.029894 ip-10-0-128-214 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 02:41:02.030070 ip-10-0-128-214 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 02:41:02.030124 ip-10-0-128-214 systemd[1]: kubelet.service: Consumed 2.083s CPU time.
Oct 30 02:41:02.046939 ip-10-0-128-214 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:41:02.123687 ip-10-0-128-214 kubenswrapper[4278]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:02.123687 ip-10-0-128-214 kubenswrapper[4278]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:41:02.123687 ip-10-0-128-214 kubenswrapper[4278]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:02.123687 ip-10-0-128-214 kubenswrapper[4278]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:02.125782 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125695    4278 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:02.125782 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125782    4278 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:02.125782 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125786    4278 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125790    4278 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125793    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125795    4278 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125798    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125802    4278 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125805    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125807    4278 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125810    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125813    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125816    4278 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125819    4278 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125822    4278 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125825    4278 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125827    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125830    4278 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125833    4278 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125836    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125838    4278 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125841    4278 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:02.125881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125844    4278 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125846    4278 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125849    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125852    4278 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125855    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125858    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125861    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125864    4278 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125867    4278 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125869    4278 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125872    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125875    4278 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125877    4278 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125880    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125883    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125886    4278 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125888    4278 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125891    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125894    4278 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:02.126370 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125897    4278 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125899    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125902    4278 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125904    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125907    4278 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125910    4278 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125912    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125915    4278 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125917    4278 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125920    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125923    4278 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125926    4278 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125928    4278 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125931    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125934    4278 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125937    4278 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125939    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125942    4278 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125945    4278 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125948    4278 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:02.126855 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125951    4278 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125954    4278 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125956    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125959    4278 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125961    4278 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125964    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125966    4278 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125969    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125971    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125974    4278 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125977    4278 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125979    4278 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125982    4278 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125984    4278 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125987    4278 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125989    4278 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125992    4278 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125995    4278 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.125998    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126000    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:02.127342 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126003    4278 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126005    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126009    4278 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126015    4278 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126019    4278 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126022    4278 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126025    4278 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126028    4278 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126031    4278 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126034    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126036    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126039    4278 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126041    4278 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126044    4278 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126047    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126049    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126052    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126055    4278 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:02.127853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126057    4278 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126060    4278 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126062    4278 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126065    4278 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126068    4278 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126072    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126074    4278 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126931    4278 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126942    4278 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126946    4278 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126950    4278 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126953    4278 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126957    4278 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126961    4278 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126965    4278 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126968    4278 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126970    4278 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126973    4278 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126976    4278 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:02.128282 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126978    4278 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126981    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126984    4278 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126987    4278 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126989    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126992    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126995    4278 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.126998    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127000    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127003    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127006    4278 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127009    4278 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127012    4278 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127014    4278 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127017    4278 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127020    4278 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127022    4278 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127025    4278 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127028    4278 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:02.128769 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127031    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127033    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127036    4278 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127038    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127041    4278 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127043    4278 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127046    4278 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127049    4278 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127054    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127057    4278 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127059    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127062    4278 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127065    4278 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127067    4278 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127070    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127072    4278 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127075    4278 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127081    4278 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127084    4278 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:02.129235 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127087    4278 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127090    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127092    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127095    4278 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127097    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127100    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127103    4278 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127105    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127108    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127111    4278 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127114    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127116    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127119    4278 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127121    4278 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127124    4278 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127126    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127129    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127132    4278 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127135    4278 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127137    4278 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:02.129723 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127140    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127142    4278 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127145    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127148    4278 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127150    4278 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127153    4278 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127156    4278 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127158    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127161    4278 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127163    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127166    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127169    4278 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127171    4278 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127174    4278 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127176    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127179    4278 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127182    4278 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127184    4278 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127186    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127189    4278 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:02.130221 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127192    4278 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127194    4278 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127197    4278 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127199    4278 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127202    4278 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127204    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127207    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127209    4278 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127212    4278 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127214    4278 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127217    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127219    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127222    4278 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127224    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127227    4278 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127231    4278 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127744    4278 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127750    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127754    4278 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127756    4278 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:02.130768 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127760    4278 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127763    4278 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127765    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127768    4278 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127771    4278 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127776    4278 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127780    4278 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127782    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127784    4278 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127787    4278 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127790    4278 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127792    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127795    4278 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127797    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127800    4278 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127803    4278 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127805    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127808    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127810    4278 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127813    4278 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:02.131265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127815    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127818    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127821    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127823    4278 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127826    4278 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127828    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127831    4278 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127834    4278 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127837    4278 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127840    4278 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127842    4278 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127845    4278 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127848    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127850    4278 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127853    4278 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127856    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127859    4278 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127863    4278 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127866    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:02.131978 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127868    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127871    4278 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127873    4278 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127876    4278 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127879    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127881    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127884    4278 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127886    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127889    4278 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127891    4278 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127894    4278 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127896    4278 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127901    4278 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127905    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127908    4278 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127910    4278 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127913    4278 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127915    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127918    4278 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:02.132813 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127920    4278 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127923    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127926    4278 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127929    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127931    4278 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127934    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127936    4278 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127939    4278 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127941    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127943    4278 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127946    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127950    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127952    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127955    4278 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127957    4278 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127960    4278 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127962    4278 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127965    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127967    4278 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:02.133654 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127974    4278 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127976    4278 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127979    4278 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127981    4278 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127984    4278 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127986    4278 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127989    4278 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127991    4278 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127994    4278 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127997    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.127999    4278 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128002    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128005    4278 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128009    4278 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128013    4278 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128016    4278 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128019    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128022    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128024    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128027    4278 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:02.134484 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128029    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128032    4278 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128034    4278 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128038    4278 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128042    4278 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128128    4278 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128135    4278 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128141    4278 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128146    4278 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128151    4278 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128154    4278 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128158    4278 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128164    4278 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128167    4278 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128170    4278 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128174    4278 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128177    4278 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128181    4278 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128184    4278 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128187    4278 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128190    4278 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128193    4278 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128196    4278 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:41:02.135319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128200    4278 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128204    4278 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128207    4278 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128210    4278 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128214    4278 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128218    4278 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128221    4278 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128225    4278 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128228    4278 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128231    4278 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128234    4278 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128237    4278 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128241    4278 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128245    4278 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128249    4278 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128252    4278 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128255    4278 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128258    4278 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128261    4278 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128264    4278 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128269    4278 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128274    4278 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128277    4278 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128281    4278 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:41:02.136192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128284    4278 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128288    4278 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128292    4278 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128295    4278 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128298    4278 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128301    4278 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128305    4278 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128308    4278 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128311    4278 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128314    4278 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128317    4278 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128320    4278 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128324    4278 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128327    4278 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128330    4278 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128335    4278 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128339    4278 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128344    4278 flags.go:64] FLAG: --help="false"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128348    4278 flags.go:64] FLAG: --hostname-override="ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128353    4278 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128358    4278 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128364    4278 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128368    4278 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:41:02.136912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128372    4278 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128375    4278 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128378    4278 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128381    4278 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128384    4278 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128387    4278 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128393    4278 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128396    4278 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128399    4278 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128402    4278 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128405    4278 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128408    4278 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128411    4278 flags.go:64] FLAG: --lock-file=""
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128414    4278 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128417    4278 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128433    4278 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128439    4278 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128442    4278 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128445    4278 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128448    4278 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128451    4278 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128454    4278 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128457    4278 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128460    4278 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128464    4278 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:41:02.137542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128468    4278 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128472    4278 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128475    4278 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128478    4278 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128481    4278 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128485    4278 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128489    4278 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128492    4278 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128495    4278 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128501    4278 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128505    4278 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128508    4278 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128511    4278 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128515    4278 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128518    4278 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128521    4278 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128524    4278 flags.go:64] FLAG: --port="10250"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128527    4278 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128530    4278 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0231e2d94c84776d2"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128533    4278 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128536    4278 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128539    4278 flags.go:64] FLAG: --register-node="true"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128542    4278 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128546    4278 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128549    4278 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128552    4278 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:41:02.138191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128555    4278 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128559    4278 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128562    4278 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128565    4278 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128568    4278 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128572    4278 flags.go:64] FLAG: --runonce="false"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128575    4278 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128578    4278 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128582    4278 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128585    4278 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128588    4278 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128592    4278 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128595    4278 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128598    4278 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128601    4278 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128604    4278 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128607    4278 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128610    4278 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128613    4278 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128617    4278 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128620    4278 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128625    4278 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128628    4278 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128631    4278 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128636    4278 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:41:02.138890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128639    4278 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128642    4278 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128645    4278 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128647    4278 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128651    4278 flags.go:64] FLAG: --v="2"
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128655    4278 flags.go:64] FLAG: --version="false"
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128659    4278 flags.go:64] FLAG: --vmodule=""
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128663    4278 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.128666    4278 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128799    4278 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128802    4278 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128805    4278 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128808    4278 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128810    4278 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128813    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128816    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128819    4278 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128821    4278 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128825    4278 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128828    4278 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128831    4278 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:02.139536 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128833    4278 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128836    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128838    4278 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128841    4278 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128844    4278 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128848    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128850    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128853    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128856    4278 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128859    4278 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128861    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128864    4278 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128867    4278 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128870    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128872    4278 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128876    4278 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128878    4278 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128881    4278 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128883    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128886    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:02.140104 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128889    4278 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128892    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128894    4278 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128897    4278 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128900    4278 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128902    4278 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128905    4278 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128907    4278 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128910    4278 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128914    4278 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128917    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128920    4278 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128923    4278 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128925    4278 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128928    4278 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128930    4278 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128933    4278 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128937    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128939    4278 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128942    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:02.140719 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128945    4278 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128947    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128950    4278 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128953    4278 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128956    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128958    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128961    4278 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128964    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128966    4278 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128969    4278 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128971    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128974    4278 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128976    4278 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128979    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128982    4278 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128984    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128987    4278 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128989    4278 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128992    4278 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:02.141323 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128994    4278 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.128998    4278 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129002    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129005    4278 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129007    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129010    4278 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129013    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129015    4278 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129019    4278 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129022    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129026    4278 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129030    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129033    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129035    4278 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129038    4278 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129041    4278 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129043    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129046    4278 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129049    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:02.141891 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129051    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129054    4278 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129056    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129059    4278 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129062    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129065    4278 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129069    4278 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129072    4278 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129075    4278 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129078    4278 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129080    4278 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129083    4278 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129086    4278 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129088    4278 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129091    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:02.142348 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129096    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.129121    4278 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129154    4278 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129159    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129163    4278 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129165    4278 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129168    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129171    4278 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129174    4278 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129177    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129180    4278 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129183    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129185    4278 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129188    4278 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129191    4278 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129194    4278 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:02.142784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129196    4278 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129199    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129201    4278 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129204    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129206    4278 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129209    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129211    4278 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129215    4278 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129219    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129222    4278 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129225    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129227    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129230    4278 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129232    4278 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129235    4278 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129238    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129242    4278 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129245    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129248    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:02.143250 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129251    4278 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129254    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129256    4278 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129259    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129262    4278 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129266    4278 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129270    4278 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129273    4278 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129275    4278 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129278    4278 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129281    4278 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129284    4278 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129286    4278 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129289    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129292    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129294    4278 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129297    4278 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129299    4278 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129302    4278 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:02.144025 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129305    4278 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129307    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129310    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129312    4278 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129315    4278 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129318    4278 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129320    4278 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129323    4278 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129325    4278 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129328    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129331    4278 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129334    4278 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129337    4278 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129340    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129342    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129345    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129348    4278 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129350    4278 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129353    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129356    4278 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:02.161819 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129359    4278 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:02.160896 ip-10-0-128-214 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129362    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129364    4278 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129367    4278 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129370    4278 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129373    4278 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129376    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129378    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129381    4278 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129383    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129386    4278 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129389    4278 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129391    4278 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129394    4278 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129397    4278 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129399    4278 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129402    4278 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129405    4278 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129408    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129410    4278 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:02.162749 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129413    4278 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129415    4278 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129431    4278 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129435    4278 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129437    4278 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129440    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129443    4278 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129445    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129448    4278 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129451    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129453    4278 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129457    4278 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129461    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.129463    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.134076    4278 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.134095    4278 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.134121    4278 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.134128    4278 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134212    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134220    4278 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:02.163383 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134225    4278 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134230    4278 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134235    4278 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134239    4278 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134244    4278 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134248    4278 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134253    4278 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134257    4278 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134261    4278 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134266    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134270    4278 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134274    4278 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134278    4278 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134283    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134287    4278 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134291    4278 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134295    4278 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134300    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134304    4278 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134308    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:02.164287 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134313    4278 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134317    4278 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134322    4278 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134327    4278 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134331    4278 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134335    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134340    4278 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134346    4278 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134350    4278 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134355    4278 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134360    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134364    4278 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134368    4278 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134373    4278 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134377    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134381    4278 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134386    4278 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134390    4278 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134394    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:02.165274 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134399    4278 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134403    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134407    4278 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134411    4278 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134416    4278 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134438    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134444    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134448    4278 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134452    4278 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134458    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134462    4278 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134466    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134470    4278 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134474    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134479    4278 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134483    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134487    4278 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134490    4278 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134494    4278 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134498    4278 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:02.166197 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134502    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134508    4278 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134512    4278 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134516    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134521    4278 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134526    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134530    4278 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134534    4278 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134539    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134543    4278 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134548    4278 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134552    4278 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134556    4278 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134561    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134566    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134570    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134574    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134578    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134583    4278 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134588    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:02.167169 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134592    4278 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134596    4278 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134601    4278 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134605    4278 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134609    4278 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134614    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134618    4278 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134622    4278 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134626    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134630    4278 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134634    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134639    4278 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134643    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134647    4278 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134652    4278 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134657    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134661    4278 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134665    4278 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134672    4278 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:02.168077 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134678    4278 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134684    4278 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134690    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134695    4278 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134700    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.134704    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.134746    4278 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135290    4278 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135583    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135590    4278 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135595    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135601    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135606    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135611    4278 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135623    4278 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:02.168966 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135628    4278 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135633    4278 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135638    4278 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135642    4278 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135647    4278 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135654    4278 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135662    4278 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135668    4278 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135674    4278 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135679    4278 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135684    4278 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135694    4278 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135712    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135718    4278 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135723    4278 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135727    4278 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135732    4278 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135737    4278 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135741    4278 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135746    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:02.169676 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135750    4278 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135755    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135760    4278 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135765    4278 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135774    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135779    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135783    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135788    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135793    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135798    4278 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135803    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135810    4278 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135816    4278 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135821    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135826    4278 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135830    4278 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135841    4278 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135847    4278 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135852    4278 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135857    4278 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:02.170538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135862    4278 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135867    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135871    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135876    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135881    4278 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135885    4278 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135892    4278 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135897    4278 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135903    4278 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135913    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135918    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135922    4278 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135927    4278 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135931    4278 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135936    4278 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135941    4278 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135945    4278 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.135995    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136020    4278 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136062    4278 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:02.171357 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136235    4278 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136241    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136244    4278 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136247    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136252    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136255    4278 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136259    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136262    4278 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136265    4278 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136267    4278 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136270    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136273    4278 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136276    4278 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136279    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136281    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136284    4278 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136287    4278 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136289    4278 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136292    4278 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:02.172194 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136294    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136297    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136300    4278 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136307    4278 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136310    4278 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136312    4278 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136315    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136320    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136322    4278 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136325    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136328    4278 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136330    4278 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136334    4278 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136337    4278 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136340    4278 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136343    4278 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136346    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136348    4278 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:02.172976 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136351    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.136386    4278 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136454    4278 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136460    4278 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136463    4278 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136466    4278 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136469    4278 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136472    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136474    4278 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136477    4278 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136480    4278 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136482    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136485    4278 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136487    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136490    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136492    4278 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:02.173652 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136496    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136499    4278 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136502    4278 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136504    4278 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136507    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136510    4278 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136513    4278 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136516    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136519    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136521    4278 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136524    4278 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136527    4278 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136530    4278 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136532    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136535    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136538    4278 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136541    4278 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136544    4278 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136546    4278 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:02.174339 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136549    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136551    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136554    4278 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136556    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136559    4278 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136562    4278 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136564    4278 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136567    4278 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136569    4278 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136572    4278 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136575    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136577    4278 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136580    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136583    4278 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136585    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136588    4278 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136591    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136593    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136596    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:02.174932 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136599    4278 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136601    4278 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136604    4278 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136607    4278 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136609    4278 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136612    4278 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136615    4278 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136619    4278 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136623    4278 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136625    4278 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136628    4278 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136632    4278 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136635    4278 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136637    4278 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136640    4278 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136643    4278 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136645    4278 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136648    4278 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136651    4278 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:02.175757 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136653    4278 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136656    4278 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136658    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136661    4278 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136664    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136666    4278 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136669    4278 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136672    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136675    4278 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136680    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136683    4278 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136685    4278 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136688    4278 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136691    4278 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136695    4278 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136699    4278 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136702    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136705    4278 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136708    4278 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136710    4278 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:02.176543 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136713    4278 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136716    4278 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136719    4278 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136721    4278 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136724    4278 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136727    4278 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136730    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136732    4278 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136735    4278 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136737    4278 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136741    4278 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136744    4278 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136746    4278 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136749    4278 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:02.136751    4278 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.136815    4278 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.137811    4278 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.137892    4278 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.138075    4278 server.go:1014] "Starting client certificate rotation"
Oct 30 02:41:02.177139 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.138225    4278 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:41:02.177953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.138274    4278 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:35:22 +0000 UTC" deadline="2028-05-10 07:43:50.131476815 +0000 UTC"
Oct 30 02:41:02.177953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.138301    4278 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13397h2m47.99317785s"
Oct 30 02:41:02.177953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.138540    4278 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:41:02.177953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.139578    4278 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:41:02.177953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.141031    4278 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:41:02.177953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.142519    4278 log.go:25] "Validated CRI v1 image API"
Oct 30 02:41:02.177953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.143230    4278 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:41:02.177953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.144700    4278 fs.go:135] Filesystem UUIDs: map[7723a7df-d15a-493d-8b60-d7065cdb0280:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a8a23d16-dbad-43b6-8e3f-2b627a766078:/dev/nvme0n1p3]
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.144721    4278 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/041e79e9268d06e9f6c0ea23703be9d2adfc7838fe0f0332b090d7d15734ad12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/041e79e9268d06e9f6c0ea23703be9d2adfc7838fe0f0332b090d7d15734ad12/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1325e70da97fd8655b5e6bb11036c1f1d8c301b583aac316d9cc5a9abeef24fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1325e70da97fd8655b5e6bb11036c1f1d8c301b583aac316d9cc5a9abeef24fb/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d3c604880f128c7d323fb77a2981a67dd60a4a757e220a4a5d8b336d6f3194d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d3c604880f128c7d323fb77a2981a67dd60a4a757e220a4a5d8b336d6f3194d/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e2b71f3d259eda648f648548cdd288d42be7e2fcf0c39ba96acd4b5908c82f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e2b71f3d259eda648f648548cdd288d42be7e2fcf0c39ba96acd4b5908c82f1/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cbeac0618c785897f48a1544b9e440850a670db635b8099fc23d239f50c0fa8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cbeac0618c785897f48a1544b9e440850a670db635b8099fc23d239f50c0fa8/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4537d2e4bc34fbe8df163fe61996e315a55601a06ae27833e882a657ca1b7c19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4537d2e4bc34fbe8df163fe61996e315a55601a06ae27833e882a657ca1b7c19/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65af3ee82b74fbe246fd1f0d8ec2c18839506aba7f7b90af65e1524ba24fe5ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65af3ee82b74fbe246fd1f0d8ec2c18839506aba7f7b90af65e1524ba24fe5ee/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85af4cb15541f4a7d77c6922e5ee34d913889cab64a8c1aced97741eafc00fb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85af4cb15541f4a7d77c6922e5ee34d913889cab64a8c1aced97741eafc00fb1/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86187fbbdeccd22c99dbcef68042190fae77aa6858f61592ba702cb20bc0d859/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86187fbbdeccd22c99dbcef68042190fae77aa6858f61592ba702cb20bc0d859/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/954fbbbee2472bb1d4ae83ca2b4de19d60c2f61c2783168d033adaa421159bd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/954fbbbee2472bb1d4ae83ca2b4de19d60c2f61c2783168d033adaa421159bd9/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b63509df8040379b61d43121cbad4b87d4f970d307681aba00cbc384f6d349f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b63509df8040379b61d43121cbad4b87d4f970d307681aba00cbc384f6d349f0/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be3a657b3f6bec0b57c5185354f0e4af4364edd118182482532eb6bdd54cdee0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be3a657b3f6bec0b57c5185354f0e4af4364edd118182482532eb6bdd54cdee0/userdata/shm major:0 minor:71 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01973b42-4064-4c7c-b6ef-b85d89b2f152/volumes/kubernetes.io~projected/kube-api-access-rk5rz:{mountpoint:/var/lib/kubelet/pods/01973b42-4064-4c7c-b6ef-b85d89b2f152/volumes/kubernetes.io~projected/kube-api-access-rk5rz major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15a18601-beef-4a52-a98f-47414f15efe1/volumes/kubernetes.io~projected/kube-api-access-cpg5c:{mountpoint:/var/lib/kubelet/pods/15a18601-beef-4a52-a98f-47414f15efe1/volumes/kubernetes.io~projected/kube-api-access-cpg5c major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bc38ad4-6aa2-4d8d-a273-d1cae373266c/volumes/kubernetes.io~projected/kube-api-access-j5wf7:{mountpoint:/var/lib/kubelet/pods/2bc38ad4-6aa2-4d8d-a273-d1cae373266c/volumes/kubernetes.io~projected/kube-api-access-j5wf7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50ae74bc-1468-4326-b1bc-c6e640b5abd4/volumes/kubernetes.io~projected/kube-api-access-bldfr:{mountpoint:/var/lib/kubelet/pods/50ae74bc-1468-4326-b1bc-c6e640b5abd4/volumes/kubernetes.io~projected/kube-api-access-bldfr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54cc321a-a26d-4aab-8e95-144a2f9aaa41/volumes/kubernetes.io~projected/kube-api-access-7schg:{mountpoint:/var/lib/kubelet/pods/54cc321a-a26d-4aab-8e95-144a2f9aaa41/volumes/kubernetes.io~projected/kube-api-access-7schg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f82d4e3-4636-4d13-9770-ae32b7719a7b/volumes/kubernetes.io~projected/kube-api-access-tm7m6:{mountpoint:/var/lib/kubelet/pods/5f82d4e3-4636-4d13-9770-ae32b7719a7b/volumes/kubernetes.io~projected/kube-api-access-tm7m6 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/702224cc-b73b-468d-82e4-1647167c6314/volumes/kubernetes.io~projected/kube-api-access-wg6g6:{mountpoint:/var/lib/kubelet/pods/702224cc-b73b-468d-82e4-1647167c6314/volumes/kubernetes.io~projected/kube-api-access-wg6g6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/729ae77b-bc29-4685-b33b-38feb7fe0a68/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/729ae77b-bc29-4685-b33b-38feb7fe0a68/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8308710a-3726-4a5e-90b5-7a6d63a1a25b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8308710a-3726-4a5e-90b5-7a6d63a1a25b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6/volumes/kubernetes.io~projected/kube-api-access-gncl2:{mountpoint:/var/lib/kubelet/pods/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6/volumes/kubernetes.io~projected/kube-api-access-gncl2 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb12906b-59f7-473c-ad0b-c87929d603e4/volumes/kubernetes.io~projected/kube-api-access-db4lq:{mountpoint:/var/lib/kubelet/pods/eb12906b-59f7-473c-ad0b-c87929d603e4/volumes/kubernetes.io~projected/kube-api-access-db4lq major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2d2d439-0d2f-4377-a043-f307916402f7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f2d2d439-0d2f-4377-a043-f307916402f7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2d2d439-0d2f-4377-a043-f307916402f7/volumes/kubernetes.io~projected/kube-api-access-qc6z9:{mountpoint:/var/lib/kubelet/pods/f2d2d439-0d2f-4377-a043-f307916402f7/volumes/kubernetes.io~projected/kube-api-access-qc6z9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2d2d439-0d2f-4377-a043-f307916402f7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f2d2d439-0d2f-4377-a043-f307916402f7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a5b383c42002cf8c21c64c6be4006707070d8c5514c3657a40b5762167e1cc3a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1c4498cc750250d39a52aed514c9a763d6fd295e52bff3ed3045a50aa66d2322/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b0be88ac4149a2797f6a48bac13948a9903999d8cbd58c254160e2745e2a7a38/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7d6e00eb3de3845339ab7eec6d260227eedd3253168e4853287f7da3df407fb1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/00539dbddc82b84001b9b086d77ba818cb49e8f17bab009a9650f0b354b894bd/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d7044f407d8a049ee5cc5ffb446ba261cc8ccf4ca91f063336edce399e236aef/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d53c14d443b361302a35e0a71b2c3a008141d1c61c314127fb00a5499bc74616/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/55ef5362e1867878057b74be3f0ad8e331c64f4c0850d870b0d2136287fe1761/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/21ee9f1ed89db9a5d67874a04fe8311c078fb1a9e09e43171a683d0a57d37c95/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/795e92dca648b2588e8efe85b239a30f04409940195a14380bdef6828bb163ae/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/4dce84a9dff1311fd7c857bb3a6ec9663bb46eda570cdd3aba1e1b519086092f/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/040f772cbbda4daddd747d6090d2272a2d01dc3b645b611002f7a52c77f604b3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6188417e640cf06e1e007a9d0f8a475eb206c23c0c807f6c5adee5fd232e7cb1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c9a02a8cf569b851da82be01da67a7cb24d267e52b07fefc7b607275d02bcf53/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8f2e6b109a1069ef1f3fd7e2a01ca19ba319da833d31b864d3cb1bfa3c40fc86/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b1eef78e012fa05e4d8c4913234a2bd6ab2f28e14a5f14cc1e47ee1bfee51ade/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d733a35454ffbef88a11bd788979e05e47ee747fe82c62e6f70408c9c17351aa/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5641a0f5c7735f532e2cbe3bbbef964e603050a16240eba6f70898c5118d8efb/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9c577fa54e605fce266711057bc807c2fe2ffdca1ef096f36a76eb1b352179a9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/10c03adacb06e01f02403958a81c180d5a4c47739f0c48adcaf0f2c85ff053c0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/253a81c65eaa52acdbdb1061c1d26002505ce55d7abe512a06c1fbc6982c3519/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/4bb979e5b20bae7ca28f7cfc06db2a48f5eaf20355173351ff7bdaabaf170d25/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/fcd5b21e1c04a4faa98c21b9831a733b1bdabfd8e976d6137de27b7296540b4e/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c6d0bcb7f47d790f5c6d33fe14006147dd63641cb4a387745edf7fadbd260fd2/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/ef2a1878fb928f6dc902d6baafa76fc0f98dd8b0c9b235f51f53d1a4739c49a5/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/53b40e71afed5fd1aa03597e9550b789161dbebb624647ee4a4de475c71075a9/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/127c83a6b23ac09275109a593293dd102867dbdac440263ea594ef2169c3b471/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/2dfabd5fbca5ae1cc0e91c4d411e09c0868b209a2ce58524f5ea30ecf1c03c51/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/80c90b5c74dd8b82bb0d5cf272c2b05e4aabb9d2567fe07372903f0d2eb3aacd/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/4079c521fa8af3df9fec10d56bfb8101ab246c61020abfff35ff9b6ff7d3471b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/17fbf7342117265530f7ef228b9ae70d685521318fa67089dd524afa362daaf3/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/bf92123c7dc5fbac71aeac50193aba81ee2a8e9f5eaa9a3d6b11123434d5b0d7/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/343d2b85a051b888ccbea3abbbbecb56f89d88d1ed90c8b59de106a3274c7299/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/ae6d3ef3ee230bef84fbde341ab59e8022e83c220c3ba436e4e3025627187115/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/7cb0621bdf32f55601675533a34ab5fbcc7b8af2c3d96878a0093e1a9e7a96f2/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.150846    4278 manager.go:217] Machine: {Timestamp:2026-10-30 02:41:02.150374604 +0000 UTC m=+0.074582785 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099521 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2476f7ae0076573aebef590271bb42 SystemUUID:ec2476f7-ae00-7657-3aeb-ef590271bb42 BootID:5c3f52f6-e09c-4a1e-a410-6c48d533fbb8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/729ae77b-bc29-4685-b33b-38feb7fe0a68/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e2b71f3d259eda648f648548cdd288d42be7e2fcf0c39ba96acd4b5908c82f1/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6/volumes/kubernetes.io~projected/kube-api-access-gncl2 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1325e70da97fd8655b5e6bb11036c1f1d8c301b583aac316d9cc5a9abeef24fb/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/954fbbbee2472bb1d4ae83ca2b4de19d60c2f61c2783168d033adaa421159bd9/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f2d2d439-0d2f-4377-a043-f307916402f7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50ae74bc-1468-4326-b1bc-c6e640b5abd4/volumes/kubernetes.io~projected/kube-api-access-bldfr DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d3c604880f128c7d323fb77a2981a67dd60a4a757e220a4a5d8b336d6f3194d/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4537d2e4bc34fbe8df163fe61996e315a55601a06ae27833e882a657ca1b7c19/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85af4cb15541f4a7d77c6922e5ee34d913889cab64a8c1aced97741eafc00fb1/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2d2d439-0d2f-4377-a043-f307916402f7/volumes/kubernetes.io~projected/kube-api-access-qc6z9 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b63509df8040379b61d43121cbad4b87d4f970d307681aba00cbc384f6d349f0/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/041e79e9268d06e9f6c0ea23703be9d2adfc7838fe0f0332b090d7d15734ad12/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cbeac0618c785897f48a1544b9e440850a670db635b8099fc23d239f50c0fa8/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2bc38ad4-6aa2-4d8d-a273-d1cae373266c/volumes/kubernetes.io~projected/kube-api-access-j5wf7 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be3a657b3f6bec0b57c5185354f0e4af4364edd118182482532eb6bdd54cdee0/userdata/shm DeviceMajor:0 DeviceMinor:71 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86187fbbdeccd22c99dbcef68042190fae77aa6858f61592ba702cb20bc0d859/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f82d4e3-4636-4d13-9770-ae32b7719a7b/volumes/kubernetes.io~projected/kube-api-access-tm7m6 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8308710a-3726-4a5e-90b5-7a6d63a1a25b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eb12906b-59f7-473c-ad0b-c87929d603e4/volumes/kubernetes.io~projected/kube-api-access-db4lq DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/15a18601-beef-4a52-a98f-47414f15efe1/volumes/kubernetes.io~projected/kube-api-access-cpg5c DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/54cc321a-a26d-4aab-8e95-144a2f9aaa41/volumes/kubernetes.io~projected/kube-api-access-7schg DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/702224cc-b73b-468d-82e4-1647167c6314/volumes/kubernetes.io~projected/kube-api-access-wg6g6 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2d2d439-0d2f-4377-a043-f307916402f7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01973b42-4064-4c7c-b6ef-b85d89b2f152/volumes/kubernetes.io~projected/kube-api-access-rk5rz DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65af3ee82b74fbe246fd1f0d8ec2c18839506aba7f7b90af65e1524ba24fe5ee/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1e2b71f3d259eda MacAddress:16:99:4c:6d:9e:8b Speed:10000 Mtu:8901} {Name:b63509df8040379 MacAddress:e6:6a:ab:28:d2:36 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c9:b4:e6:a5 Speed:0 Mtu:9001} {Name:br-int MacAddress:7a:ce:ad:30:e9:d3 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:c9:b4:e6:a5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:2e:dc:81:e2:e5:13 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:4f:62:b1:ae:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151069    4278 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151115    4278 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151378    4278 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151395    4278 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-214.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151543    4278 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151552    4278 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151563    4278 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151582    4278 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151666    4278 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151821    4278 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151938    4278 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151973    4278 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:41:02.178386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151986    4278 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.151996    4278 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.152006    4278 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.152015    4278 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.153203    4278 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.153219    4278 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.154214    4278 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156393    4278 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156402    4278 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156519    4278 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156700    4278 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156727    4278 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156814    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156826    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156832    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156838    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156844    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156850    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156855    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156860    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156867    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156873    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156882    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156891    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156924    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.156930    4278 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.160248    4278 server.go:1263] "Started kubelet"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.160301    4278 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.160352    4278 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.160441    4278 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.160739    4278 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.161458    4278 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.163587    4278 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.163599    4278 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.163654    4278 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:24 +0000 UTC" deadline="2028-04-01 12:29:32.405133197 +0000 UTC"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.163717    4278 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.163730    4278 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.163831    4278 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.163687    4278 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12465h48m30.241448912s"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:02.164237    4278 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-214.ec2.internal\" not found"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.164873    4278 factory.go:55] Registering systemd factory
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.164892    4278 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165258    4278 factory.go:153] Registering CRI-O factory
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165271    4278 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165333    4278 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165352    4278 factory.go:103] Registering Raw factory
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165368    4278 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165613    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bc38ad4-6aa2-4d8d-a273-d1cae373266c" volumeName="kubernetes.io/projected/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-kube-api-access-j5wf7" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165647    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54cc321a-a26d-4aab-8e95-144a2f9aaa41" volumeName="kubernetes.io/projected/54cc321a-a26d-4aab-8e95-144a2f9aaa41-kube-api-access-7schg" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165661    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f82d4e3-4636-4d13-9770-ae32b7719a7b" volumeName="kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165673    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f82d4e3-4636-4d13-9770-ae32b7719a7b" volumeName="kubernetes.io/projected/5f82d4e3-4636-4d13-9770-ae32b7719a7b-kube-api-access-tm7m6" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165685    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="702224cc-b73b-468d-82e4-1647167c6314" volumeName="kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-multus-daemon-config" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165698    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb12906b-59f7-473c-ad0b-c87929d603e4" volumeName="kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165713    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ae74bc-1468-4326-b1bc-c6e640b5abd4" volumeName="kubernetes.io/projected/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kube-api-access-bldfr" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165725    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f82d4e3-4636-4d13-9770-ae32b7719a7b" volumeName="kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165733    4278 manager.go:319] Starting recovery of all containers
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165742    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8308710a-3726-4a5e-90b5-7a6d63a1a25b" volumeName="kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165754    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6" volumeName="kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-tmp" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165766    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d2d439-0d2f-4377-a043-f307916402f7" volumeName="kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-env-overrides" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165780    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d2d439-0d2f-4377-a043-f307916402f7" volumeName="kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165792    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d2d439-0d2f-4377-a043-f307916402f7" volumeName="kubernetes.io/projected/f2d2d439-0d2f-4377-a043-f307916402f7-kube-api-access-qc6z9" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165803    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d2d439-0d2f-4377-a043-f307916402f7" volumeName="kubernetes.io/secret/f2d2d439-0d2f-4377-a043-f307916402f7-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165819    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01973b42-4064-4c7c-b6ef-b85d89b2f152" volumeName="kubernetes.io/empty-dir/01973b42-4064-4c7c-b6ef-b85d89b2f152-tmp-dir" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165830    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="729ae77b-bc29-4685-b33b-38feb7fe0a68" volumeName="kubernetes.io/configmap/729ae77b-bc29-4685-b33b-38feb7fe0a68-konnectivity-ca" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165842    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6" volumeName="kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-tuned" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165857    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a18601-beef-4a52-a98f-47414f15efe1" volumeName="kubernetes.io/projected/15a18601-beef-4a52-a98f-47414f15efe1-kube-api-access-cpg5c" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165869    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54cc321a-a26d-4aab-8e95-144a2f9aaa41" volumeName="kubernetes.io/configmap/54cc321a-a26d-4aab-8e95-144a2f9aaa41-serviceca" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165881    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f82d4e3-4636-4d13-9770-ae32b7719a7b" volumeName="kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165892    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="702224cc-b73b-468d-82e4-1647167c6314" volumeName="kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165904    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="702224cc-b73b-468d-82e4-1647167c6314" volumeName="kubernetes.io/projected/702224cc-b73b-468d-82e4-1647167c6314-kube-api-access-wg6g6" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165915    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="729ae77b-bc29-4685-b33b-38feb7fe0a68" volumeName="kubernetes.io/secret/729ae77b-bc29-4685-b33b-38feb7fe0a68-agent-certs" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165938    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6" volumeName="kubernetes.io/projected/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-kube-api-access-gncl2" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165954    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d2d439-0d2f-4377-a043-f307916402f7" volumeName="kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-config" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165966    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01973b42-4064-4c7c-b6ef-b85d89b2f152" volumeName="kubernetes.io/projected/01973b42-4064-4c7c-b6ef-b85d89b2f152-kube-api-access-rk5rz" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165979    4278 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15a18601-beef-4a52-a98f-47414f15efe1" volumeName="kubernetes.io/configmap/15a18601-beef-4a52-a98f-47414f15efe1-iptables-alerter-script" seLinuxMountContext=""
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165990    4278 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:41:02.179408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.165998    4278 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:41:02.189590 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.189552    4278 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:41:02.190914 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.190894    4278 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:41:02.190914 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.190916    4278 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:41:02.191106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.190936    4278 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:41:02.191106 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:02.190974    4278 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 02:41:02.210853 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.210826    4278 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 02:41:02.212530 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.212182    4278 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:02.212530 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.212233    4278 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:02.221855 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.221804    4278 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="3736a450088187de1cd822378b7b3b629a69d43264d639e9a9d5eca9405713a4" exitCode=0
Oct 30 02:41:02.221855 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.221853    4278 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="c697a308cd9ba26f1627e4f27c73a53f1ba187523a64c8c09ba442182b29da64" exitCode=0
Oct 30 02:41:02.221855 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.221863    4278 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="d25a5a69e3b4e7aa45375553ab8b2a999925e4987094cf7b2331c9d7e141df94" exitCode=0
Oct 30 02:41:02.222178 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.221868    4278 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="478d264a9262f2a2861032d5c0ec4562f18cd2649ebe5c4d6e54aa9092e5363b" exitCode=0
Oct 30 02:41:02.222178 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.221877    4278 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="deb09f6110c575fa5d1c0b6833652b7fc4a057f3c9ec2a82291b1752fda79299" exitCode=0
Oct 30 02:41:02.222178 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.221885    4278 generic.go:356] "Generic (PLEG): container finished" podID="5f82d4e3-4636-4d13-9770-ae32b7719a7b" containerID="0ebed9e7fa5622697d7bb29a2394adc51f178ea6c66c8c74d251659fb211cc9b" exitCode=0
Oct 30 02:41:02.224224 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.224189    4278 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:02.224512 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.224493    4278 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:02.224755 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.224734    4278 generic.go:356] "Generic (PLEG): container finished" podID="72498d2be80a2b536a0b73a7eab91abc" containerID="85a7cc084fbd7e6eb17b5886a8a8ca1cccbe1ce920b5cfcc0be86db02849f967" exitCode=0
Oct 30 02:41:02.228512 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.228493    4278 manager.go:324] Recovery completed
Oct 30 02:41:02.242440 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242403    4278 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:41:02.242440 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242417    4278 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:41:02.242626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242453    4278 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:41:02.242626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242571    4278 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 02:41:02.242626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242582    4278 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 02:41:02.242626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242599    4278 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 02:41:02.242626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242605    4278 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 02:41:02.242626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242610    4278 policy_none.go:50] "Start"
Oct 30 02:41:02.242626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242617    4278 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:41:02.242626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242626    4278 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:41:02.242855 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242747    4278 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 02:41:02.242855 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242760    4278 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 02:41:02.242855 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.242766    4278 policy_none.go:44] "Start"
Oct 30 02:41:02.246744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.246717    4278 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:41:02.249352 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.249338    4278 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:41:02.249816 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:02.249800    4278 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:41:02.249816 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.249814    4278 server.go:87] "Starting device plugin registration server"
Oct 30 02:41:02.250055 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.250044    4278 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:41:02.250112 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.250055    4278 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:41:02.250176 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.250156    4278 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:41:02.250400 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.250251    4278 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:41:02.250400 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.250290    4278 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:41:02.250400 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.250295    4278 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:41:02.250400 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.250333    4278 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:41:02.250280144Z","UUID":"8ce1fc3d-e14d-40f7-ab20-9cb70b19ab15","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:41:02.251741 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:02.251476    4278 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:41:02.252591 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.252571    4278 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:41:02.252719 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.252700    4278 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:41:02.291437 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.291324    4278 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"]
Oct 30 02:41:02.291852 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.291807    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal" event={"ID":"4b5a364f76591020026198c5396db417","Type":"ContainerStarted","Data":"cb5cac9fa8d20ea32d27f6c8b20978009aa4053c9feb5905b86f0d70ee0ed80f"}
Oct 30 02:41:02.291964 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.291860    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal" event={"ID":"4b5a364f76591020026198c5396db417","Type":"ContainerStarted","Data":"4537d2e4bc34fbe8df163fe61996e315a55601a06ae27833e882a657ca1b7c19"}
Oct 30 02:41:02.291964 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.291907    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal" event={"ID":"72498d2be80a2b536a0b73a7eab91abc","Type":"ContainerStarted","Data":"9e7a62ac7609a3498b1d64d28af7dee52dd38731c248adbcbcebc334f097f86c"}
Oct 30 02:41:02.291964 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.291925    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal" event={"ID":"72498d2be80a2b536a0b73a7eab91abc","Type":"ContainerDied","Data":"85a7cc084fbd7e6eb17b5886a8a8ca1cccbe1ce920b5cfcc0be86db02849f967"}
Oct 30 02:41:02.291964 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.291939    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal" event={"ID":"72498d2be80a2b536a0b73a7eab91abc","Type":"ContainerStarted","Data":"041e79e9268d06e9f6c0ea23703be9d2adfc7838fe0f0332b090d7d15734ad12"}
Oct 30 02:41:02.291964 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.291946    4278 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.292123 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.292021    4278 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.324464 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.324419    4278 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:02.324609 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:02.324503    4278 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-214.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.325680 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.325668    4278 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:02.325730 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:02.325699    4278 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.350897 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.350873    4278 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:02.352566 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.352549    4278 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:02.352651 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.352584    4278 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:02.352651 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.352596    4278 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-214.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:02.352651 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.352624    4278 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.390322 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.389823    4278 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.390322 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.389899    4278 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.412836 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.412808    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.412959 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.412840    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.412959 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.412860    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b5a364f76591020026198c5396db417-config\") pod \"kube-apiserver-proxy-ip-10-0-128-214.ec2.internal\" (UID: \"4b5a364f76591020026198c5396db417\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.513685 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.513649    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b5a364f76591020026198c5396db417-config\") pod \"kube-apiserver-proxy-ip-10-0-128-214.ec2.internal\" (UID: \"4b5a364f76591020026198c5396db417\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.513685 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.513677    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b5a364f76591020026198c5396db417-config\") pod \"kube-apiserver-proxy-ip-10-0-128-214.ec2.internal\" (UID: \"4b5a364f76591020026198c5396db417\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.513907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.513745    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.513907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.513769    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.513907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.513816    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:02.513907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:02.513826    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72498d2be80a2b536a0b73a7eab91abc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" (UID: \"72498d2be80a2b536a0b73a7eab91abc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:03.153097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.153058    4278 apiserver.go:52] "Watching apiserver"
Oct 30 02:41:03.191885 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.191851    4278 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:41:03.192209 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.192192    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-d9gmb","openshift-image-registry/node-ca-28mns","kube-system/global-pull-secret-syncer-wttwd","openshift-cluster-node-tuning-operator/tuned-ds5x7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal","openshift-multus/multus-additional-cni-plugins-snstr","openshift-network-operator/iptables-alerter-tdp5t","kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr","openshift-dns/node-resolver-76z2q","openshift-network-diagnostics/network-check-target-k684d","openshift-ovn-kubernetes/ovnkube-node-lxf8j","kube-system/konnectivity-agent-crnh2","openshift-multus/multus-nlhw8"]
Oct 30 02:41:03.194951 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.194931    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:41:03.207883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.207859    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:03.207883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.207865    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ph87b\""
Oct 30 02:41:03.208089 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.207914    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qc2sc\""
Oct 30 02:41:03.208089 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208019    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:41:03.208089 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208035    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r7gc2\""
Oct 30 02:41:03.208279 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208258    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:41:03.208360 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208258    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kk2sv\""
Oct 30 02:41:03.208360 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208270    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:03.208360 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208301    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:41:03.208360 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208294    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:41:03.208360 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208344    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:41:03.208360 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208319    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:03.208584 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208322    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:41:03.208584 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.208313    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:03.218113 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.218093    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:41:03.221602 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.221587    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:41:03.221930 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.221917    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:41:03.221987 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.221946    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:41:03.222184 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.222171    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j5lkg\""
Oct 30 02:41:03.222286 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.222271    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5k7wg\""
Oct 30 02:41:03.222372 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.222277    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4pml\""
Oct 30 02:41:03.222722 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.222699    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:41:03.222722 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.222719    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gf5d6\""
Oct 30 02:41:03.222929 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.222817    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:41:03.223344 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.223311    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:41:03.223654 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.223589    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:41:03.223741 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.223680    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:41:03.223802 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.223737    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:41:03.223802 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.223773    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:41:03.223902 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.223594    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:41:03.223902 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.223862    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hhdzd\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224003    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224016    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224079    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224006    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224104    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224147    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224255    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xsmz\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224309    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224386    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-86l9w\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224413    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224466    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:41:03.224500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.224482    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-prmc8\""
Oct 30 02:41:03.227803 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.227786    4278 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:03.227884 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.227848    4278 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:03.265228 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.265199    4278 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:41:03.275654 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.275633    4278 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:03.275730 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:03.275685    4278 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-214.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:03.275730 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.275711    4278 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:03.275844 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:03.275760    4278 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-214.ec2.internal"
Oct 30 02:41:03.318670 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.318642    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-k8s-cni-cncf-io\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.318670 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.318671    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-var-lib-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.318834 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.318691    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-system-cni-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.318834 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.318711    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/729ae77b-bc29-4685-b33b-38feb7fe0a68-konnectivity-ca\") pod \"konnectivity-agent-crnh2\" (UID: \"729ae77b-bc29-4685-b33b-38feb7fe0a68\") " pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:41:03.318834 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.318783    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpg5c\" (UniqueName: \"kubernetes.io/projected/15a18601-beef-4a52-a98f-47414f15efe1-kube-api-access-cpg5c\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:41:03.318976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.318867    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-systemd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.318976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.318960    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-etc-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.318976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.318962    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/729ae77b-bc29-4685-b33b-38feb7fe0a68-konnectivity-ca\") pod \"konnectivity-agent-crnh2\" (UID: \"729ae77b-bc29-4685-b33b-38feb7fe0a68\") " pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:41:03.319085 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319016    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.319085 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319043    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.319172 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319084    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:41:03.319172 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319115    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.319172 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319139    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-host\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.319172 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319167    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54cc321a-a26d-4aab-8e95-144a2f9aaa41-serviceca\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:41:03.319333 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319186    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-systemd-units\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.319333 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319244    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-netns\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.319333 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319290    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-log-socket\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.319333 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319321    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-systemd\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.319541 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319346    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-run\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.319541 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319385    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.319541 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319446    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-node-log\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.319541 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319482    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.319541 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319526    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15a18601-beef-4a52-a98f-47414f15efe1-iptables-alerter-script\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:41:03.319724 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319569    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-lib-modules\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.319724 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319598    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-tuned\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.319724 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319655    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-cnibin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.319724 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319672    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-tuned\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.319724 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319693    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01973b42-4064-4c7c-b6ef-b85d89b2f152-tmp-dir\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:41:03.319892 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319737    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-multus-certs\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.319892 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319782    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6g6\" (UniqueName: \"kubernetes.io/projected/702224cc-b73b-468d-82e4-1647167c6314-kube-api-access-wg6g6\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.319892 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319799    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01973b42-4064-4c7c-b6ef-b85d89b2f152-tmp-dir\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:41:03.319892 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319817    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15a18601-beef-4a52-a98f-47414f15efe1-iptables-alerter-script\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:41:03.319892 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319847    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-env-overrides\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.319892 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319872    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54cc321a-a26d-4aab-8e95-144a2f9aaa41-serviceca\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:41:03.320060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.319909    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:41:03.320060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320002    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-kubelet-config\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:41:03.320060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320028    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldfr\" (UniqueName: \"kubernetes.io/projected/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kube-api-access-bldfr\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.320060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320045    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15a18601-beef-4a52-a98f-47414f15efe1-host-slash\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:41:03.320060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320045    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-env-overrides\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.320198 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320148    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-conf\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.320198 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320181    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-multus-daemon-config\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.320258 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320204    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-kubelet\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.320258 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320232    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/729ae77b-bc29-4685-b33b-38feb7fe0a68-agent-certs\") pod \"konnectivity-agent-crnh2\" (UID: \"729ae77b-bc29-4685-b33b-38feb7fe0a68\") " pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:41:03.320338 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320316    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01973b42-4064-4c7c-b6ef-b85d89b2f152-hosts-file\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:41:03.320445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320417    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-conf-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.320478 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320455    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-multus-daemon-config\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.320478 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320468    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54cc321a-a26d-4aab-8e95-144a2f9aaa41-host\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:41:03.320551 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320490    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-kubelet\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.320551 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320509    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-netd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.320551 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320540    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/729ae77b-bc29-4685-b33b-38feb7fe0a68-agent-certs\") pod \"konnectivity-agent-crnh2\" (UID: \"729ae77b-bc29-4685-b33b-38feb7fe0a68\") " pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:41:03.320638 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320560    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.320638 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320611    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-sys-fs\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.320638 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320627    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-bin\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.320725 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320654    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-sys\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.320725 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320683    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-slash\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.320725 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320703    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-ovn\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.320811 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320738    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-config\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.320811 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320770    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2d2d439-0d2f-4377-a043-f307916402f7-ovn-node-metrics-cert\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.320875 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320837    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-tmp\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.320950 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320933    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-config\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.320991 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320956    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-tmp\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.320991 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320940    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-netns\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.321097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.320989    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-etc-kubernetes\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.321097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321011    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cnibin\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.321097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321027    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.321182 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321102    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2d2d439-0d2f-4377-a043-f307916402f7-ovn-node-metrics-cert\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.321182 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321131    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm7m6\" (UniqueName: \"kubernetes.io/projected/5f82d4e3-4636-4d13-9770-ae32b7719a7b-kube-api-access-tm7m6\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.321182 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321155    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5rz\" (UniqueName: \"kubernetes.io/projected/01973b42-4064-4c7c-b6ef-b85d89b2f152-kube-api-access-rk5rz\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:41:03.321182 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321166    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.321182 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321171    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-device-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.321329 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321215    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:41:03.321329 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321261    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncl2\" (UniqueName: \"kubernetes.io/projected/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-kube-api-access-gncl2\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.321329 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321305    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-socket-dir-parent\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.321456 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321438    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7schg\" (UniqueName: \"kubernetes.io/projected/54cc321a-a26d-4aab-8e95-144a2f9aaa41-kube-api-access-7schg\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:41:03.321502 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321462    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8308710a-3726-4a5e-90b5-7a6d63a1a25b-original-pull-secret\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:41:03.321502 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321487    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.321626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321610    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-socket-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.321655 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321641    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-registration-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.321689 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321665    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-system-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.321719 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321703    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.321747 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321720    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.321747 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321730    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-modprobe-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.321801 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321745    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysconfig\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.321801 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321760    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-kubernetes\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.321801 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321775    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-var-lib-kubelet\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.321878 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321800    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-bin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.321878 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321838    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-script-lib\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.321878 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321865    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-dbus\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:41:03.321965 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321888    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-cni-binary-copy\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.321965 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321905    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-multus\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.321965 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321939    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-hostroot\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.322048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.321966    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.322048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.322011    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2d2d439-0d2f-4377-a043-f307916402f7-ovnkube-script-lib\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.322048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.322011    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6z9\" (UniqueName: \"kubernetes.io/projected/f2d2d439-0d2f-4377-a043-f307916402f7-kube-api-access-qc6z9\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.322128 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.322047    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.322128 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.322045    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/702224cc-b73b-468d-82e4-1647167c6314-cni-binary-copy\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.322128 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.322085    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5wf7\" (UniqueName: \"kubernetes.io/projected/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-kube-api-access-j5wf7\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:41:03.322207 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.322190    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.335674 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.335652    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpg5c\" (UniqueName: \"kubernetes.io/projected/15a18601-beef-4a52-a98f-47414f15efe1-kube-api-access-cpg5c\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:41:03.336243 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.336227    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6g6\" (UniqueName: \"kubernetes.io/projected/702224cc-b73b-468d-82e4-1647167c6314-kube-api-access-wg6g6\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.336528 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.336512    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldfr\" (UniqueName: \"kubernetes.io/projected/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kube-api-access-bldfr\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.336870 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.336854    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4lq\" (UniqueName: \"kubernetes.io/projected/eb12906b-59f7-473c-ad0b-c87929d603e4-kube-api-access-db4lq\") pod \"network-check-target-k684d\" (UID: \"eb12906b-59f7-473c-ad0b-c87929d603e4\") " pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:41:03.337373 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.337359    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm7m6\" (UniqueName: \"kubernetes.io/projected/5f82d4e3-4636-4d13-9770-ae32b7719a7b-kube-api-access-tm7m6\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.337695 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.337679    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncl2\" (UniqueName: \"kubernetes.io/projected/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-kube-api-access-gncl2\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.340101 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.340077    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5wf7\" (UniqueName: \"kubernetes.io/projected/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-kube-api-access-j5wf7\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:41:03.340101 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.340090    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7schg\" (UniqueName: \"kubernetes.io/projected/54cc321a-a26d-4aab-8e95-144a2f9aaa41-kube-api-access-7schg\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:41:03.340636 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.340620    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6z9\" (UniqueName: \"kubernetes.io/projected/f2d2d439-0d2f-4377-a043-f307916402f7-kube-api-access-qc6z9\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.340681 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.340634    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk5rz\" (UniqueName: \"kubernetes.io/projected/01973b42-4064-4c7c-b6ef-b85d89b2f152-kube-api-access-rk5rz\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:41:03.422643 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422561    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-systemd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422643 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422592    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-etc-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422643 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422607    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422643 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422623    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422683    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-etc-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422690    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422710    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422743    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422769    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-host\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422694    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422786    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422683    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-systemd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422793    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-systemd-units\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422825    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-systemd-units\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422817    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-host\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422842    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-netns\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422871    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-run-netns\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.422883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422875    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-log-socket\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422898    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-log-socket\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422906    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-systemd\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422931    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-run\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422954    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422981    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-node-log\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.422993    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-systemd\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423004    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-run\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423005    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423027    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423047    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423038    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-lib-modules\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423049    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-node-log\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423084    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-cnibin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423101    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-multus-certs\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423110    4278 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423127    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-kubelet-config\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423138    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-cnibin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423151    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15a18601-beef-4a52-a98f-47414f15efe1-host-slash\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423161    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-multus-certs\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423171    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-conf\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423192    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-kubelet\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423194    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15a18601-beef-4a52-a98f-47414f15efe1-host-slash\") pod \"iptables-alerter-tdp5t\" (UID: \"15a18601-beef-4a52-a98f-47414f15efe1\") " pod="openshift-network-operator/iptables-alerter-tdp5t"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423207    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-kubelet-config\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423230    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-kubelet\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423233    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-lib-modules\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423216    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01973b42-4064-4c7c-b6ef-b85d89b2f152-hosts-file\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423256    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01973b42-4064-4c7c-b6ef-b85d89b2f152-hosts-file\") pod \"node-resolver-76z2q\" (UID: \"01973b42-4064-4c7c-b6ef-b85d89b2f152\") " pod="openshift-dns/node-resolver-76z2q"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423280    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-conf-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423312    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54cc321a-a26d-4aab-8e95-144a2f9aaa41-host\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423321    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-conf\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423329    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-kubelet\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423337    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-conf-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423354    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-netd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423356    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-kubelet\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.423355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423379    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423386    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54cc321a-a26d-4aab-8e95-144a2f9aaa41-host\") pod \"node-ca-28mns\" (UID: \"54cc321a-a26d-4aab-8e95-144a2f9aaa41\") " pod="openshift-image-registry/node-ca-28mns"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423397    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-sys-fs\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423414    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-bin\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423450    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423458    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-sys-fs\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423466    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-sys\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423386    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-netd\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423489    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-cni-bin\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423509    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-sys\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423491    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-slash\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423526    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-host-slash\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423537    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-ovn\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423559    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-netns\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423574    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-etc-kubernetes\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423588    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cnibin\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423597    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-ovn\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423612    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-device-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423639    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-etc-kubernetes\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423650    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-netns\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423657    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-socket-dir-parent\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423670    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-cnibin\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423697    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-device-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423684    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-socket-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423733    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-multus-socket-dir-parent\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423736    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-socket-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423737    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-registration-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423768    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50ae74bc-1468-4326-b1bc-c6e640b5abd4-registration-dir\") pod \"aws-ebs-csi-driver-node-mf5nr\" (UID: \"50ae74bc-1468-4326-b1bc-c6e640b5abd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5nr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423784    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-system-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.423801    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424035    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysctl-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424247    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-system-cni-dir\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424245    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-modprobe-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424272    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysconfig\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424286    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-kubernetes\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424300    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-var-lib-kubelet\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424313    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-bin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424326    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-dbus\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424335    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-modprobe-d\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424344    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-multus\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424342    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-kubernetes\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424373    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-etc-sysconfig\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424381    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-bin\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424384    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-var-lib-cni-multus\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424382    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6-var-lib-kubelet\") pod \"tuned-ds5x7\" (UID: \"c7fe1d12-9e94-4e41-8490-dc7f65b5a1a6\") " pod="openshift-cluster-node-tuning-operator/tuned-ds5x7"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424401    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-hostroot\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424418    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8308710a-3726-4a5e-90b5-7a6d63a1a25b-dbus\") pod \"global-pull-secret-syncer-wttwd\" (UID: \"8308710a-3726-4a5e-90b5-7a6d63a1a25b\") " pod="kube-system/global-pull-secret-syncer-wttwd"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424449    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424453    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-hostroot\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424472    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-run-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424477    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-k8s-cni-cncf-io\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424499    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/702224cc-b73b-468d-82e4-1647167c6314-host-run-k8s-cni-cncf-io\") pod \"multus-nlhw8\" (UID: \"702224cc-b73b-468d-82e4-1647167c6314\") " pod="openshift-multus/multus-nlhw8"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424502    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-var-lib-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424524    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-system-cni-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424546    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2d2d439-0d2f-4377-a043-f307916402f7-var-lib-openvswitch\") pod \"ovnkube-node-lxf8j\" (UID: \"f2d2d439-0d2f-4377-a043-f307916402f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:03.424616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.424566    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f82d4e3-4636-4d13-9770-ae32b7719a7b-system-cni-dir\") pod \"multus-additional-cni-plugins-snstr\" (UID: \"5f82d4e3-4636-4d13-9770-ae32b7719a7b\") " pod="openshift-multus/multus-additional-cni-plugins-snstr"
Oct 30 02:41:03.426079 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.425356    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc38ad4-6aa2-4d8d-a273-d1cae373266c-metrics-certs\") pod \"network-metrics-daemon-d9gmb\" (UID: \"2bc38ad4-6aa2-4d8d-a273-d1cae373266c\") " pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:41:03.495361 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.495326    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9gmb"
Oct 30 02:41:03.809128 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.809090    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d9gmb"]
Oct 30 02:41:03.812881 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:03.812835    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc38ad4_6aa2_4d8d_a273_d1cae373266c.slice/crio-1635fa4ad32d8da07e903b880be0bc52c100a2f0c9e063a170a5b69b9cedefe0 WatchSource:0}: Error finding container 1635fa4ad32d8da07e903b880be0bc52c100a2f0c9e063a170a5b69b9cedefe0: Status 404 returned error can't find the container with id 1635fa4ad32d8da07e903b880be0bc52c100a2f0c9e063a170a5b69b9cedefe0
Oct 30 02:41:03.814868 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:03.814852    4278 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:41:04.231394 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:04.231353    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d9gmb" event={"ID":"2bc38ad4-6aa2-4d8d-a273-d1cae373266c","Type":"ContainerStarted","Data":"1635fa4ad32d8da07e903b880be0bc52c100a2f0c9e063a170a5b69b9cedefe0"}
Oct 30 02:41:05.235394 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:05.235359    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d9gmb" event={"ID":"2bc38ad4-6aa2-4d8d-a273-d1cae373266c","Type":"ContainerStarted","Data":"7083bdbdbeb43b1bedd27f46f63b923644bc35c748669caf7d80879ce42efb6d"}
Oct 30 02:41:05.416445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:05.416355    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:06.239155 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:06.239114    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d9gmb" event={"ID":"2bc38ad4-6aa2-4d8d-a273-d1cae373266c","Type":"ContainerStarted","Data":"3b95ba3f6736798b559448bd2a9bef7d33a587d8e928637db846e5e2a26f08b2"}
Oct 30 02:41:10.414191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:10.414159    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:10.414581 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:10.414338    4278 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:10.586916 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:10.586886    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:41:12.311982 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.311953    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f4d98d8c-r5k8z"]
Oct 30 02:41:12.313823 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.313800    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.315875 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.315851    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-g6q8h"]
Oct 30 02:41:12.317780 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.317762    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-g6q8h"
Oct 30 02:41:12.339993 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.339972    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 02:41:12.340191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.340176    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 02:41:12.341304 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.341290    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 02:41:12.341554 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.341543    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 02:41:12.342103 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.342070    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 02:41:12.342322 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.342306    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rt4nr"]
Oct 30 02:41:12.343896 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.343879    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rt4nr"
Oct 30 02:41:12.351871 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.351854    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 02:41:12.351946 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.351873    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 02:41:12.351996 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.351980    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fhbtn\""
Oct 30 02:41:12.352982 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.352967    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lmqlv"]
Oct 30 02:41:12.355386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.355365    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-g6q8h"]
Oct 30 02:41:12.355501 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.355400    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f4d98d8c-r5k8z"]
Oct 30 02:41:12.355501 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.355484    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.369532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.369515    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:41:12.369641 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.369623    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:41:12.370057 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.370041    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qs2sf"]
Oct 30 02:41:12.371658 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.371642    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.373775 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.373758    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rt4nr"]
Oct 30 02:41:12.393370 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.393351    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmqlv"]
Oct 30 02:41:12.393628 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.393613    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:41:12.393797 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.393784    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:41:12.393857 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.393804    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:41:12.393934 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.393918    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:41:12.408065 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.408047    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qs2sf"]
Oct 30 02:41:12.408721 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.408702    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd"]
Oct 30 02:41:12.410470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.410449    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd"
Oct 30 02:41:12.438993 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.438972    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd"]
Oct 30 02:41:12.442294 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.442271    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xn2tj\""
Oct 30 02:41:12.442534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.442513    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bnjgk\""
Oct 30 02:41:12.442534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.442530    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:41:12.443073 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.443055    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 02:41:12.443204 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.443143    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:41:12.443353 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.443166    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:41:12.443471 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.443302    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xcg2x\""
Oct 30 02:41:12.443564 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.443353    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kdxrj\""
Oct 30 02:41:12.443631 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.443366    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rd75s\""
Oct 30 02:41:12.462955 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.462923    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32b8af70-8f28-4fae-adf8-c3928acc9fee-tmp\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.462955 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.462947    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23f02d0e-4210-4186-a2be-d4242f17bf1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.463109 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.462964    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c16167c-eef6-4dc3-85e7-d4a419212e43-tmp\") pod \"downloads-6dd789c8b7-g6q8h\" (UID: \"9c16167c-eef6-4dc3-85e7-d4a419212e43\") " pod="openshift-console/downloads-6dd789c8b7-g6q8h"
Oct 30 02:41:12.463109 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.462979    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-serving-cert\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.463109 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.462996    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-config\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.463109 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463066    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bncpd\" (UniqueName: \"kubernetes.io/projected/7aeb9190-4121-4fd8-a52d-5a270e004d07-kube-api-access-bncpd\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.463109 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463092    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aeb9190-4121-4fd8-a52d-5a270e004d07-config-volume\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.463256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463151    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7vh\" (UniqueName: \"kubernetes.io/projected/9c16167c-eef6-4dc3-85e7-d4a419212e43-kube-api-access-cx7vh\") pod \"downloads-6dd789c8b7-g6q8h\" (UID: \"9c16167c-eef6-4dc3-85e7-d4a419212e43\") " pod="openshift-console/downloads-6dd789c8b7-g6q8h"
Oct 30 02:41:12.463256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463174    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-oauth-config\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.463256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463191    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlfr\" (UniqueName: \"kubernetes.io/projected/32b8af70-8f28-4fae-adf8-c3928acc9fee-kube-api-access-tvlfr\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.463256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463208    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf7d6c82-7dde-42d8-b013-120eebeed803-cert\") pod \"ingress-canary-rt4nr\" (UID: \"bf7d6c82-7dde-42d8-b013-120eebeed803\") " pod="openshift-ingress-canary/ingress-canary-rt4nr"
Oct 30 02:41:12.463256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463225    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-oauth-serving-cert\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.463256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463249    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aeb9190-4121-4fd8-a52d-5a270e004d07-metrics-tls\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.463445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463306    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23f02d0e-4210-4186-a2be-d4242f17bf1d-data-volume\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.463445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463366    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aeb9190-4121-4fd8-a52d-5a270e004d07-tmp-dir\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.463445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463384    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z56f2\" (UniqueName: \"kubernetes.io/projected/bf7d6c82-7dde-42d8-b013-120eebeed803-kube-api-access-z56f2\") pod \"ingress-canary-rt4nr\" (UID: \"bf7d6c82-7dde-42d8-b013-120eebeed803\") " pod="openshift-ingress-canary/ingress-canary-rt4nr"
Oct 30 02:41:12.463445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463399    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23f02d0e-4210-4186-a2be-d4242f17bf1d-crio-socket\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.463445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463444    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23f02d0e-4210-4186-a2be-d4242f17bf1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.463445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463461    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5h7b\" (UniqueName: \"kubernetes.io/projected/23f02d0e-4210-4186-a2be-d4242f17bf1d-kube-api-access-z5h7b\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.463644 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.463506    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-service-ca\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.564515 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564456    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5h7b\" (UniqueName: \"kubernetes.io/projected/23f02d0e-4210-4186-a2be-d4242f17bf1d-kube-api-access-z5h7b\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.564515 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564486    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-service-ca\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.564515 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564507    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32b8af70-8f28-4fae-adf8-c3928acc9fee-tmp\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.564744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564522    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23f02d0e-4210-4186-a2be-d4242f17bf1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.564744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564544    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c16167c-eef6-4dc3-85e7-d4a419212e43-tmp\") pod \"downloads-6dd789c8b7-g6q8h\" (UID: \"9c16167c-eef6-4dc3-85e7-d4a419212e43\") " pod="openshift-console/downloads-6dd789c8b7-g6q8h"
Oct 30 02:41:12.564744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564566    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-serving-cert\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.564744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564592    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-config\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.564744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564614    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bncpd\" (UniqueName: \"kubernetes.io/projected/7aeb9190-4121-4fd8-a52d-5a270e004d07-kube-api-access-bncpd\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.564744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564643    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aeb9190-4121-4fd8-a52d-5a270e004d07-config-volume\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.564744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564666    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7vh\" (UniqueName: \"kubernetes.io/projected/9c16167c-eef6-4dc3-85e7-d4a419212e43-kube-api-access-cx7vh\") pod \"downloads-6dd789c8b7-g6q8h\" (UID: \"9c16167c-eef6-4dc3-85e7-d4a419212e43\") " pod="openshift-console/downloads-6dd789c8b7-g6q8h"
Oct 30 02:41:12.564744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564686    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-oauth-config\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.564744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564707    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlfr\" (UniqueName: \"kubernetes.io/projected/32b8af70-8f28-4fae-adf8-c3928acc9fee-kube-api-access-tvlfr\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.564744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564731    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf7d6c82-7dde-42d8-b013-120eebeed803-cert\") pod \"ingress-canary-rt4nr\" (UID: \"bf7d6c82-7dde-42d8-b013-120eebeed803\") " pod="openshift-ingress-canary/ingress-canary-rt4nr"
Oct 30 02:41:12.565219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564753    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-oauth-serving-cert\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.565219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564774    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aeb9190-4121-4fd8-a52d-5a270e004d07-metrics-tls\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.565219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564799    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f68b6a91-9c7c-4628-87ac-3c681f70c7e0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mjczd\" (UID: \"f68b6a91-9c7c-4628-87ac-3c681f70c7e0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd"
Oct 30 02:41:12.565219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564831    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23f02d0e-4210-4186-a2be-d4242f17bf1d-data-volume\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.565219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564860    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aeb9190-4121-4fd8-a52d-5a270e004d07-tmp-dir\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.565219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564888    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z56f2\" (UniqueName: \"kubernetes.io/projected/bf7d6c82-7dde-42d8-b013-120eebeed803-kube-api-access-z56f2\") pod \"ingress-canary-rt4nr\" (UID: \"bf7d6c82-7dde-42d8-b013-120eebeed803\") " pod="openshift-ingress-canary/ingress-canary-rt4nr"
Oct 30 02:41:12.565219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564914    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23f02d0e-4210-4186-a2be-d4242f17bf1d-crio-socket\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.565219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.564955    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23f02d0e-4210-4186-a2be-d4242f17bf1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.565219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.565051    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32b8af70-8f28-4fae-adf8-c3928acc9fee-tmp\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.565690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.565236    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23f02d0e-4210-4186-a2be-d4242f17bf1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.565690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.565370    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-service-ca\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.565690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.565509    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aeb9190-4121-4fd8-a52d-5a270e004d07-config-volume\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.565690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.565551    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-config\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.565690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.565568    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c16167c-eef6-4dc3-85e7-d4a419212e43-tmp\") pod \"downloads-6dd789c8b7-g6q8h\" (UID: \"9c16167c-eef6-4dc3-85e7-d4a419212e43\") " pod="openshift-console/downloads-6dd789c8b7-g6q8h"
Oct 30 02:41:12.565942 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.565767    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aeb9190-4121-4fd8-a52d-5a270e004d07-tmp-dir\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.565942 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.565876    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23f02d0e-4210-4186-a2be-d4242f17bf1d-crio-socket\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.566068 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.566041    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23f02d0e-4210-4186-a2be-d4242f17bf1d-data-volume\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.566163 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.566144    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-oauth-serving-cert\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.567780 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.567740    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aeb9190-4121-4fd8-a52d-5a270e004d07-metrics-tls\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.567885 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.567840    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf7d6c82-7dde-42d8-b013-120eebeed803-cert\") pod \"ingress-canary-rt4nr\" (UID: \"bf7d6c82-7dde-42d8-b013-120eebeed803\") " pod="openshift-ingress-canary/ingress-canary-rt4nr"
Oct 30 02:41:12.567885 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.567847    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23f02d0e-4210-4186-a2be-d4242f17bf1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.567885 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.567875    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-oauth-config\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.568710 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.568689    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-serving-cert\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.585535 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.585511    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bncpd\" (UniqueName: \"kubernetes.io/projected/7aeb9190-4121-4fd8-a52d-5a270e004d07-kube-api-access-bncpd\") pod \"dns-default-lmqlv\" (UID: \"7aeb9190-4121-4fd8-a52d-5a270e004d07\") " pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.585609 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.585532    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5h7b\" (UniqueName: \"kubernetes.io/projected/23f02d0e-4210-4186-a2be-d4242f17bf1d-kube-api-access-z5h7b\") pod \"insights-runtime-extractor-qs2sf\" (UID: \"23f02d0e-4210-4186-a2be-d4242f17bf1d\") " pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.588144 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.588125    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z56f2\" (UniqueName: \"kubernetes.io/projected/bf7d6c82-7dde-42d8-b013-120eebeed803-kube-api-access-z56f2\") pod \"ingress-canary-rt4nr\" (UID: \"bf7d6c82-7dde-42d8-b013-120eebeed803\") " pod="openshift-ingress-canary/ingress-canary-rt4nr"
Oct 30 02:41:12.588352 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.588335    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlfr\" (UniqueName: \"kubernetes.io/projected/32b8af70-8f28-4fae-adf8-c3928acc9fee-kube-api-access-tvlfr\") pod \"console-7f4d98d8c-r5k8z\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") " pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.589591 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.589573    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7vh\" (UniqueName: \"kubernetes.io/projected/9c16167c-eef6-4dc3-85e7-d4a419212e43-kube-api-access-cx7vh\") pod \"downloads-6dd789c8b7-g6q8h\" (UID: \"9c16167c-eef6-4dc3-85e7-d4a419212e43\") " pod="openshift-console/downloads-6dd789c8b7-g6q8h"
Oct 30 02:41:12.625356 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.625332    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:12.629952 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.629926    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-g6q8h"
Oct 30 02:41:12.653315 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.653292    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rt4nr"
Oct 30 02:41:12.666043 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.666012    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f68b6a91-9c7c-4628-87ac-3c681f70c7e0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mjczd\" (UID: \"f68b6a91-9c7c-4628-87ac-3c681f70c7e0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd"
Oct 30 02:41:12.667821 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.667636    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:12.683496 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.682957    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f68b6a91-9c7c-4628-87ac-3c681f70c7e0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mjczd\" (UID: \"f68b6a91-9c7c-4628-87ac-3c681f70c7e0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd"
Oct 30 02:41:12.683496 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.683086    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qs2sf"
Oct 30 02:41:12.726306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.725599    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd"
Oct 30 02:41:12.958889 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.958817    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-g6q8h"]
Oct 30 02:41:12.961620 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.961594    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f4d98d8c-r5k8z"]
Oct 30 02:41:12.964738 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:12.964711    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c16167c_eef6_4dc3_85e7_d4a419212e43.slice/crio-862cb21e78c547d5faf00e224cb303db8543beb8fa39622cac2ee9d3c3233ae5 WatchSource:0}: Error finding container 862cb21e78c547d5faf00e224cb303db8543beb8fa39622cac2ee9d3c3233ae5: Status 404 returned error can't find the container with id 862cb21e78c547d5faf00e224cb303db8543beb8fa39622cac2ee9d3c3233ae5
Oct 30 02:41:12.973781 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:12.973763    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rt4nr"]
Oct 30 02:41:12.978120 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:12.978095    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7d6c82_7dde_42d8_b013_120eebeed803.slice/crio-a6d3e9b11ec5802dfed0911db12c9558a1e935a896bb1699da52a4ca4f1e6840 WatchSource:0}: Error finding container a6d3e9b11ec5802dfed0911db12c9558a1e935a896bb1699da52a4ca4f1e6840: Status 404 returned error can't find the container with id a6d3e9b11ec5802dfed0911db12c9558a1e935a896bb1699da52a4ca4f1e6840
Oct 30 02:41:13.041165 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:13.041139    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qs2sf"]
Oct 30 02:41:13.041516 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:13.041506    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd"]
Oct 30 02:41:13.041889 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:13.041871    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmqlv"]
Oct 30 02:41:13.047095 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:13.047073    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68b6a91_9c7c_4628_87ac_3c681f70c7e0.slice/crio-578815f6c31f275e9b34be74e722a9a2d3feac1ffb1d410929b149fb6fad03b3 WatchSource:0}: Error finding container 578815f6c31f275e9b34be74e722a9a2d3feac1ffb1d410929b149fb6fad03b3: Status 404 returned error can't find the container with id 578815f6c31f275e9b34be74e722a9a2d3feac1ffb1d410929b149fb6fad03b3
Oct 30 02:41:13.047841 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:13.047816    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f02d0e_4210_4186_a2be_d4242f17bf1d.slice/crio-ba5d6111131e84b1291e9a8e9d482caad70013117a91508dbdbac6e3a63ab89b WatchSource:0}: Error finding container ba5d6111131e84b1291e9a8e9d482caad70013117a91508dbdbac6e3a63ab89b: Status 404 returned error can't find the container with id ba5d6111131e84b1291e9a8e9d482caad70013117a91508dbdbac6e3a63ab89b
Oct 30 02:41:13.048633 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:13.048572    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aeb9190_4121_4fd8_a52d_5a270e004d07.slice/crio-7becb807cf60291913e64ed39204cf583b965b415f7b8642858b2e8ebc407ee6 WatchSource:0}: Error finding container 7becb807cf60291913e64ed39204cf583b965b415f7b8642858b2e8ebc407ee6: Status 404 returned error can't find the container with id 7becb807cf60291913e64ed39204cf583b965b415f7b8642858b2e8ebc407ee6
Oct 30 02:41:13.256221 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:13.256187    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd" event={"ID":"f68b6a91-9c7c-4628-87ac-3c681f70c7e0","Type":"ContainerStarted","Data":"578815f6c31f275e9b34be74e722a9a2d3feac1ffb1d410929b149fb6fad03b3"}
Oct 30 02:41:13.257314 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:13.257294    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmqlv" event={"ID":"7aeb9190-4121-4fd8-a52d-5a270e004d07","Type":"ContainerStarted","Data":"7becb807cf60291913e64ed39204cf583b965b415f7b8642858b2e8ebc407ee6"}
Oct 30 02:41:13.258459 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:13.258415    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rt4nr" event={"ID":"bf7d6c82-7dde-42d8-b013-120eebeed803","Type":"ContainerStarted","Data":"a6d3e9b11ec5802dfed0911db12c9558a1e935a896bb1699da52a4ca4f1e6840"}
Oct 30 02:41:13.260029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:13.260010    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qs2sf" event={"ID":"23f02d0e-4210-4186-a2be-d4242f17bf1d","Type":"ContainerStarted","Data":"92c2349848e3cd2a7859614543cacc5f84f4b175a4c4af6a0a7e8cce587f6af4"}
Oct 30 02:41:13.260132 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:13.260051    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qs2sf" event={"ID":"23f02d0e-4210-4186-a2be-d4242f17bf1d","Type":"ContainerStarted","Data":"ba5d6111131e84b1291e9a8e9d482caad70013117a91508dbdbac6e3a63ab89b"}
Oct 30 02:41:13.261095 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:13.261074    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-g6q8h" event={"ID":"9c16167c-eef6-4dc3-85e7-d4a419212e43","Type":"ContainerStarted","Data":"862cb21e78c547d5faf00e224cb303db8543beb8fa39622cac2ee9d3c3233ae5"}
Oct 30 02:41:13.262199 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:13.262177    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4d98d8c-r5k8z" event={"ID":"32b8af70-8f28-4fae-adf8-c3928acc9fee","Type":"ContainerStarted","Data":"641904dc9541d6b353ee3b20e588343dd08908a00789d4349ae0e2400cfbcb47"}
Oct 30 02:41:14.268795 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:14.268695    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qs2sf" event={"ID":"23f02d0e-4210-4186-a2be-d4242f17bf1d","Type":"ContainerStarted","Data":"c36461e9abfc07ca0bab2f0e1b0148375c8013b1ea5d0fe0c357efeabbb39837"}
Oct 30 02:41:15.276763 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:15.276682    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd" event={"ID":"f68b6a91-9c7c-4628-87ac-3c681f70c7e0","Type":"ContainerStarted","Data":"fce71bb8717255d3625bb8499c0781a40859bbfce297d1ec0d7e3d985363563a"}
Oct 30 02:41:15.277769 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:15.277704    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd"
Oct 30 02:41:15.284534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:15.284483    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd"
Oct 30 02:41:15.352326 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:15.351662    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mjczd" podStartSLOduration=1.854999345 podStartE2EDuration="3.351649134s" totalImagesPullingTime="1.496649789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:15.315983689 +0000 UTC m=+13.240191879" watchObservedRunningTime="2026-10-30 02:41:15.351649134 +0000 UTC m=+13.275857321"
Oct 30 02:41:16.282732 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:16.282646    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmqlv" event={"ID":"7aeb9190-4121-4fd8-a52d-5a270e004d07","Type":"ContainerStarted","Data":"b8262ab3aaf603e27081fca0bdc293b0db4ea56fa77caced5117a6b7f88e0ea1"}
Oct 30 02:41:16.282732 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:16.282716    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmqlv" event={"ID":"7aeb9190-4121-4fd8-a52d-5a270e004d07","Type":"ContainerStarted","Data":"bf6876c4d3e8e8e3728f64277cac4c76e429db6b8c815eef8866feddf10741af"}
Oct 30 02:41:16.283514 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:16.283493    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:16.288331 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:16.288267    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qs2sf" event={"ID":"23f02d0e-4210-4186-a2be-d4242f17bf1d","Type":"ContainerStarted","Data":"0c05424423c1d59756c95cca03c8a70259e743bf157b5bcc5e9b71a946e3cd5c"}
Oct 30 02:41:16.321141 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:16.321086    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-lmqlv" podStartSLOduration=1.602962732 podStartE2EDuration="4.321067216s" totalImagesPullingTime="2.718104484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:16.320636486 +0000 UTC m=+14.244844675" watchObservedRunningTime="2026-10-30 02:41:16.321067216 +0000 UTC m=+14.245275385"
Oct 30 02:41:16.361073 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:16.361021    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qs2sf" podStartSLOduration=1.533062081 podStartE2EDuration="4.36100781s" totalImagesPullingTime="2.827945729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:16.360680478 +0000 UTC m=+14.284888679" watchObservedRunningTime="2026-10-30 02:41:16.36100781 +0000 UTC m=+14.285215997"
Oct 30 02:41:17.294634 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:17.294119    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rt4nr" event={"ID":"bf7d6c82-7dde-42d8-b013-120eebeed803","Type":"ContainerStarted","Data":"032ebdab3395d3b10065f0e7b907291e7aead77746f61aabfdfebfd40cf27ca5"}
Oct 30 02:41:17.333714 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:17.333659    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rt4nr" podStartSLOduration=1.305430884 podStartE2EDuration="5.33364568s" totalImagesPullingTime="4.028214796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:17.332668615 +0000 UTC m=+15.256876802" watchObservedRunningTime="2026-10-30 02:41:17.33364568 +0000 UTC m=+15.257853859"
Oct 30 02:41:18.299303 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:18.299237    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4d98d8c-r5k8z" event={"ID":"32b8af70-8f28-4fae-adf8-c3928acc9fee","Type":"ContainerStarted","Data":"09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896"}
Oct 30 02:41:18.339824 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:18.339763    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f4d98d8c-r5k8z" podStartSLOduration=2.073275823 podStartE2EDuration="6.339748043s" totalImagesPullingTime="4.26647222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:18.339649075 +0000 UTC m=+16.263857261" watchObservedRunningTime="2026-10-30 02:41:18.339748043 +0000 UTC m=+16.263956231"
Oct 30 02:41:19.747443 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.747384    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66f654b9fd-pb7sp"]
Oct 30 02:41:19.750173 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.750148    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.805967 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.805737    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 02:41:19.806829 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.806762    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f654b9fd-pb7sp"]
Oct 30 02:41:19.830765 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.830731    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-config\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.830917 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.830771    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-trusted-ca-bundle\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.830917 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.830797    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c6ccb04-6783-4b87-b648-8ab732e4764b-tmp\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.830917 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.830813    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-serving-cert\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.830917 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.830879    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-service-ca\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.831106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.831081    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-oauth-serving-cert\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.831234 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.831129    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84kc9\" (UniqueName: \"kubernetes.io/projected/0c6ccb04-6783-4b87-b648-8ab732e4764b-kube-api-access-84kc9\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.831234 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.831207    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-oauth-config\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.932266 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.932043    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-oauth-serving-cert\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.932266 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.932112    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84kc9\" (UniqueName: \"kubernetes.io/projected/0c6ccb04-6783-4b87-b648-8ab732e4764b-kube-api-access-84kc9\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.932266 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.932146    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-oauth-config\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.932266 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.932187    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-config\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.932266 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.932216    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-trusted-ca-bundle\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.932266 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.932248    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c6ccb04-6783-4b87-b648-8ab732e4764b-tmp\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.932266 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.932274    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-serving-cert\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.932677 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.932300    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-service-ca\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.932944 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.932917    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-oauth-serving-cert\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.933060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.932965    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-service-ca\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.933203 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.933181    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-config\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.933256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.933204    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-trusted-ca-bundle\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.933291 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.933273    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c6ccb04-6783-4b87-b648-8ab732e4764b-tmp\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.934896 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.934872    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-oauth-config\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.935370 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.935347    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-serving-cert\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:19.954222 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:19.954197    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84kc9\" (UniqueName: \"kubernetes.io/projected/0c6ccb04-6783-4b87-b648-8ab732e4764b-kube-api-access-84kc9\") pod \"console-66f654b9fd-pb7sp\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") " pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:20.063139 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.063092    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:20.306534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.306505    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-25bsh"]
Oct 30 02:41:20.309843 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.309821    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.334784 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.334753    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"]
Oct 30 02:41:20.336842 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.336815    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:41:20.336969 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.336888    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 02:41:20.337046 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.336996    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:41:20.337046 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.337009    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.337046 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.337023    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 02:41:20.337046 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.337031    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 02:41:20.337046 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.337034    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vvql5\""
Oct 30 02:41:20.337488 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.337470    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 02:41:20.360084 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.360054    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4whnw\""
Oct 30 02:41:20.360084 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.360060    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 02:41:20.360537 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.360519    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 02:41:20.365501 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.365476    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"]
Oct 30 02:41:20.389570 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.389535    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f654b9fd-pb7sp"]
Oct 30 02:41:20.396017 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:20.395960    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6ccb04_6783_4b87_b648_8ab732e4764b.slice/crio-93683146c33b9ff4bfbba5f29d76149e30bfbd4235e88010e1110384f1ea39dc WatchSource:0}: Error finding container 93683146c33b9ff4bfbba5f29d76149e30bfbd4235e88010e1110384f1ea39dc: Status 404 returned error can't find the container with id 93683146c33b9ff4bfbba5f29d76149e30bfbd4235e88010e1110384f1ea39dc
Oct 30 02:41:20.427445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.427401    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"]
Oct 30 02:41:20.431095 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.431066    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.435770 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.435724    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c803dbce-4c1c-4d64-a85c-fc766855d54a-metrics-client-ca\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.435911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.435769    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.435911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.435799    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c803dbce-4c1c-4d64-a85c-fc766855d54a-root\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.435911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.435825    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.435911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.435860    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-accelerators-collector-config\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.435911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.435900    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-wtmp\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.436106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.435949    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4br6n\" (UniqueName: \"kubernetes.io/projected/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-kube-api-access-4br6n\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.436106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.435982    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c803dbce-4c1c-4d64-a85c-fc766855d54a-sys\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.436106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.436007    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-textfile\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.436106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.436032    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhqr\" (UniqueName: \"kubernetes.io/projected/c803dbce-4c1c-4d64-a85c-fc766855d54a-kube-api-access-jfhqr\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.436106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.436059    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.436106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.436086    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.436261 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.436110    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-tls\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.466591 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.466551    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 02:41:20.466819 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.466796    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 02:41:20.466965 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.466923    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-59mr7\""
Oct 30 02:41:20.472061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.472031    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"]
Oct 30 02:41:20.509962 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.509932    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 02:41:20.537338 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537304    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c803dbce-4c1c-4d64-a85c-fc766855d54a-sys\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.537338 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537347    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-textfile\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.537568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537369    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhqr\" (UniqueName: \"kubernetes.io/projected/c803dbce-4c1c-4d64-a85c-fc766855d54a-kube-api-access-jfhqr\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.537568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537389    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.537568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537418    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.537568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537454    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.537568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537480    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.537568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537511    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-tls\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.537568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537540    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c803dbce-4c1c-4d64-a85c-fc766855d54a-metrics-client-ca\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.537568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537564    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.537910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537589    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c803dbce-4c1c-4d64-a85c-fc766855d54a-root\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.537910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537615    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.537910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537647    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12f3a6be-b590-4482-b69c-2429c37d0dff-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.537910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537678    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-accelerators-collector-config\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.537910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537712    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-wtmp\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.537910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537727    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.537910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537754    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/12f3a6be-b590-4482-b69c-2429c37d0dff-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.537910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537775    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66npw\" (UniqueName: \"kubernetes.io/projected/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-api-access-66npw\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.537910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.537804    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4br6n\" (UniqueName: \"kubernetes.io/projected/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-kube-api-access-4br6n\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.538249 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.538207    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c803dbce-4c1c-4d64-a85c-fc766855d54a-sys\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.538544 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.538522    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-textfile\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.540417 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.539337    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-wtmp\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.540417 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.539568    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c803dbce-4c1c-4d64-a85c-fc766855d54a-root\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.540417 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.539816    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c803dbce-4c1c-4d64-a85c-fc766855d54a-metrics-client-ca\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.540417 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.539853    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.540417 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.539877    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-accelerators-collector-config\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.541745 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.541722    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-tls\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.541993 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.541974    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c803dbce-4c1c-4d64-a85c-fc766855d54a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.542864 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.542820    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.543267 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.543202    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.561746 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.561707    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4br6n\" (UniqueName: \"kubernetes.io/projected/435fb283-f5e3-4a4c-85a1-f64ab00e90b3-kube-api-access-4br6n\") pod \"openshift-state-metrics-7bd7489957-k574r\" (UID: \"435fb283-f5e3-4a4c-85a1-f64ab00e90b3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.564542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.564515    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhqr\" (UniqueName: \"kubernetes.io/projected/c803dbce-4c1c-4d64-a85c-fc766855d54a-kube-api-access-jfhqr\") pod \"node-exporter-25bsh\" (UID: \"c803dbce-4c1c-4d64-a85c-fc766855d54a\") " pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.624164 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.624082    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-25bsh"
Oct 30 02:41:20.635563 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:20.635520    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc803dbce_4c1c_4d64_a85c_fc766855d54a.slice/crio-9eab736b80c98fb0de4f0a7e3d518fa6247362076138df405b1b80d1bad7013c WatchSource:0}: Error finding container 9eab736b80c98fb0de4f0a7e3d518fa6247362076138df405b1b80d1bad7013c: Status 404 returned error can't find the container with id 9eab736b80c98fb0de4f0a7e3d518fa6247362076138df405b1b80d1bad7013c
Oct 30 02:41:20.638521 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.638497    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.638651 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.638537    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/12f3a6be-b590-4482-b69c-2429c37d0dff-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.638651 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.638563    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66npw\" (UniqueName: \"kubernetes.io/projected/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-api-access-66npw\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.638651 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.638607    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.638651 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.638649    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.638914 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.638684    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12f3a6be-b590-4482-b69c-2429c37d0dff-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.638970 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.638941    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/12f3a6be-b590-4482-b69c-2429c37d0dff-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.639642 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.639622    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.639853 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.639789    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12f3a6be-b590-4482-b69c-2429c37d0dff-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.641505 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.641478    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.641621 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.641563    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.651441 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.651395    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"
Oct 30 02:41:20.661579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.661550    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66npw\" (UniqueName: \"kubernetes.io/projected/12f3a6be-b590-4482-b69c-2429c37d0dff-kube-api-access-66npw\") pod \"kube-state-metrics-5dc8d7d89b-bhd9g\" (UID: \"12f3a6be-b590-4482-b69c-2429c37d0dff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.754011 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.753978    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"
Oct 30 02:41:20.972979 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:20.972905    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-k574r"]
Oct 30 02:41:20.976767 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:20.976728    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435fb283_f5e3_4a4c_85a1_f64ab00e90b3.slice/crio-7217e8816f35f2b10c5b9bf0412b4a14183bda3137be5abdc395b8226054eb94 WatchSource:0}: Error finding container 7217e8816f35f2b10c5b9bf0412b4a14183bda3137be5abdc395b8226054eb94: Status 404 returned error can't find the container with id 7217e8816f35f2b10c5b9bf0412b4a14183bda3137be5abdc395b8226054eb94
Oct 30 02:41:21.093247 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:21.092591    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g"]
Oct 30 02:41:21.103481 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:21.103334    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f3a6be_b590_4482_b69c_2429c37d0dff.slice/crio-243ea50b12e2d8e83a3f66c1fb1096e58ceca54d9523c94e7e96bba8470e853d WatchSource:0}: Error finding container 243ea50b12e2d8e83a3f66c1fb1096e58ceca54d9523c94e7e96bba8470e853d: Status 404 returned error can't find the container with id 243ea50b12e2d8e83a3f66c1fb1096e58ceca54d9523c94e7e96bba8470e853d
Oct 30 02:41:21.285899 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:21.285867    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lxf8j"
Oct 30 02:41:21.311503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:21.311467    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g" event={"ID":"12f3a6be-b590-4482-b69c-2429c37d0dff","Type":"ContainerStarted","Data":"243ea50b12e2d8e83a3f66c1fb1096e58ceca54d9523c94e7e96bba8470e853d"}
Oct 30 02:41:21.313994 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:21.313967    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r" event={"ID":"435fb283-f5e3-4a4c-85a1-f64ab00e90b3","Type":"ContainerStarted","Data":"54367565fdccd21bf1db750f693614ce750b8b3467197672342411e43e34b1e2"}
Oct 30 02:41:21.314137 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:21.314021    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r" event={"ID":"435fb283-f5e3-4a4c-85a1-f64ab00e90b3","Type":"ContainerStarted","Data":"c810389332bbfdcddfecb2c1b91ed4a1e0c81b0b25fa26a5e2498b613e8fdff2"}
Oct 30 02:41:21.314137 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:21.314049    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r" event={"ID":"435fb283-f5e3-4a4c-85a1-f64ab00e90b3","Type":"ContainerStarted","Data":"7217e8816f35f2b10c5b9bf0412b4a14183bda3137be5abdc395b8226054eb94"}
Oct 30 02:41:21.315646 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:21.315620    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25bsh" event={"ID":"c803dbce-4c1c-4d64-a85c-fc766855d54a","Type":"ContainerStarted","Data":"9eab736b80c98fb0de4f0a7e3d518fa6247362076138df405b1b80d1bad7013c"}
Oct 30 02:41:21.317579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:21.317549    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f654b9fd-pb7sp" event={"ID":"0c6ccb04-6783-4b87-b648-8ab732e4764b","Type":"ContainerStarted","Data":"d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985"}
Oct 30 02:41:21.317688 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:21.317597    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f654b9fd-pb7sp" event={"ID":"0c6ccb04-6783-4b87-b648-8ab732e4764b","Type":"ContainerStarted","Data":"93683146c33b9ff4bfbba5f29d76149e30bfbd4235e88010e1110384f1ea39dc"}
Oct 30 02:41:22.325170 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.324643    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r" event={"ID":"435fb283-f5e3-4a4c-85a1-f64ab00e90b3","Type":"ContainerStarted","Data":"949694db5cce3b84f8394a3c0b8291c4c56083d906dd71273e77f58c07c8deea"}
Oct 30 02:41:22.326567 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.326533    4278 generic.go:356] "Generic (PLEG): container finished" podID="c803dbce-4c1c-4d64-a85c-fc766855d54a" containerID="2b5a8a8dbb6252be35643f2c7c454737777a4b93e68187e058b72c2e7b43d576" exitCode=0
Oct 30 02:41:22.326679 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.326626    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25bsh" event={"ID":"c803dbce-4c1c-4d64-a85c-fc766855d54a","Type":"ContainerDied","Data":"2b5a8a8dbb6252be35643f2c7c454737777a4b93e68187e058b72c2e7b43d576"}
Oct 30 02:41:22.372851 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.372798    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k574r" podStartSLOduration=1.346186323 podStartE2EDuration="2.372781848s" totalImagesPullingTime="1.026595525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:22.37240298 +0000 UTC m=+20.296611171" watchObservedRunningTime="2026-10-30 02:41:22.372781848 +0000 UTC m=+20.296990039"
Oct 30 02:41:22.373264 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.373224    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66f654b9fd-pb7sp" podStartSLOduration=3.373218458 podStartE2EDuration="3.373218458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:21.427292448 +0000 UTC m=+19.351500638" watchObservedRunningTime="2026-10-30 02:41:22.373218458 +0000 UTC m=+20.297426647"
Oct 30 02:41:22.625729 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.625691    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:22.625959 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.625886    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:22.631488 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.631437    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:22.635630 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.635608    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7675f749d8-tlnp5"]
Oct 30 02:41:22.638814 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.638793    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.674025 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.673749    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e1mha3nn9dvo6\""
Oct 30 02:41:22.674025 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.673855    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 02:41:22.674025 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.673947    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 02:41:22.674025 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.673749    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 02:41:22.674025 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.673950    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gtgxt\""
Oct 30 02:41:22.674025 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.673859    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 02:41:22.674025 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.674000    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 02:41:22.697134 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.697018    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7675f749d8-tlnp5"]
Oct 30 02:41:22.756622 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.756505    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bc64762-491a-420d-aabe-3c8e280bf1ea-metrics-client-ca\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.756622 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.756555    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.756859 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.756699    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bc64762-491a-420d-aabe-3c8e280bf1ea-tmp\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.756859 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.756758    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.756859 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.756795    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-tls\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.756859 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.756833    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.757081 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.756885    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llqwt\" (UniqueName: \"kubernetes.io/projected/5bc64762-491a-420d-aabe-3c8e280bf1ea-kube-api-access-llqwt\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.757081 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.756971    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-grpc-tls\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.757081 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.757020    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.857792 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.857746    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llqwt\" (UniqueName: \"kubernetes.io/projected/5bc64762-491a-420d-aabe-3c8e280bf1ea-kube-api-access-llqwt\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.857976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.857813    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-grpc-tls\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.857976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.857834    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.857976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.857863    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bc64762-491a-420d-aabe-3c8e280bf1ea-metrics-client-ca\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.857976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.857886    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.857976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.857910    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bc64762-491a-420d-aabe-3c8e280bf1ea-tmp\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.857976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.857940    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.857976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.857971    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-tls\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.858361 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.857996    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.858497 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.858472    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bc64762-491a-420d-aabe-3c8e280bf1ea-tmp\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.858779 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.858752    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bc64762-491a-420d-aabe-3c8e280bf1ea-metrics-client-ca\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.861185 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.861106    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.861185 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.861130    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-grpc-tls\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.861185 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.861139    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.861356 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.861239    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-tls\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.861932 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.861910    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.863099 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.863080    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bc64762-491a-420d-aabe-3c8e280bf1ea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.891163 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.891130    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llqwt\" (UniqueName: \"kubernetes.io/projected/5bc64762-491a-420d-aabe-3c8e280bf1ea-kube-api-access-llqwt\") pod \"thanos-querier-7675f749d8-tlnp5\" (UID: \"5bc64762-491a-420d-aabe-3c8e280bf1ea\") " pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:22.960907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:22.960819    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:23.295226 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:23.295185    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7675f749d8-tlnp5"]
Oct 30 02:41:23.303063 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:23.303026    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc64762_491a_420d_aabe_3c8e280bf1ea.slice/crio-bc68cee101cc14da1e932b8cdd3b0b17d2ee7b4c4257089f566ec6dd785e07c2 WatchSource:0}: Error finding container bc68cee101cc14da1e932b8cdd3b0b17d2ee7b4c4257089f566ec6dd785e07c2: Status 404 returned error can't find the container with id bc68cee101cc14da1e932b8cdd3b0b17d2ee7b4c4257089f566ec6dd785e07c2
Oct 30 02:41:23.333653 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:23.333613    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g" event={"ID":"12f3a6be-b590-4482-b69c-2429c37d0dff","Type":"ContainerStarted","Data":"7d9ffb34a64437b7f5ce5e8cac468e67384587297867b89d2c59135e5030ef8f"}
Oct 30 02:41:23.334142 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:23.333693    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g" event={"ID":"12f3a6be-b590-4482-b69c-2429c37d0dff","Type":"ContainerStarted","Data":"25a102c40985f6de3bf62ba7d0969874792509d63fb6374dc08529471d879f70"}
Oct 30 02:41:23.334142 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:23.333726    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g" event={"ID":"12f3a6be-b590-4482-b69c-2429c37d0dff","Type":"ContainerStarted","Data":"32c4049a5d04ab2df36de6ae877e320d92c76d58c6e8d2662c481bdc5a7ab70e"}
Oct 30 02:41:23.336228 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:23.336193    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25bsh" event={"ID":"c803dbce-4c1c-4d64-a85c-fc766855d54a","Type":"ContainerStarted","Data":"8237f5aec5aeedfc76a35adb9f0afd5f61d6836d5b6d16cdf2dd362c6979cece"}
Oct 30 02:41:23.336374 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:23.336252    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25bsh" event={"ID":"c803dbce-4c1c-4d64-a85c-fc766855d54a","Type":"ContainerStarted","Data":"0f87d79d41b7f66461b1cba5d50302c4fa3e69587578514170d25c89502c4af5"}
Oct 30 02:41:23.337570 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:23.337535    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5" event={"ID":"5bc64762-491a-420d-aabe-3c8e280bf1ea","Type":"ContainerStarted","Data":"bc68cee101cc14da1e932b8cdd3b0b17d2ee7b4c4257089f566ec6dd785e07c2"}
Oct 30 02:41:23.343173 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:23.343148    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:41:23.375087 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:23.375011    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bhd9g" podStartSLOduration=1.99514014 podStartE2EDuration="3.374990094s" totalImagesPullingTime="1.379849954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:23.374143203 +0000 UTC m=+21.298351392" watchObservedRunningTime="2026-10-30 02:41:23.374990094 +0000 UTC m=+21.299198282"
Oct 30 02:41:23.495390 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:23.495336    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-25bsh" podStartSLOduration=2.439455789 podStartE2EDuration="3.495322272s" totalImagesPullingTime="981.627541ms" totalInitContainerRuntime="74.238942ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:23.49429732 +0000 UTC m=+21.418505511" watchObservedRunningTime="2026-10-30 02:41:23.495322272 +0000 UTC m=+21.419530496"
Oct 30 02:41:24.194982 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:24.194947    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:41:24.320888 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:24.320854    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-crnh2"
Oct 30 02:41:25.350253 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.350123    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5" event={"ID":"5bc64762-491a-420d-aabe-3c8e280bf1ea","Type":"ContainerStarted","Data":"a3580c8d56cd66bf8b3ed350646316dd8b5a7a0aa04a7d2cc6cadec5fe7d514c"}
Oct 30 02:41:25.350253 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.350198    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5" event={"ID":"5bc64762-491a-420d-aabe-3c8e280bf1ea","Type":"ContainerStarted","Data":"355913d172624c971770b91f5fdae22272b12394c7d4aa5af7ec1f93a7beefe1"}
Oct 30 02:41:25.741161 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.741118    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-bcf4486-xh5ht"]
Oct 30 02:41:25.745950 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.744995    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.760673 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.760646    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rrdlk\""
Oct 30 02:41:25.772569 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.772535    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 02:41:25.772749 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.772623    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 02:41:25.772749 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.772719    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 02:41:25.772861 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.772797    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 02:41:25.772910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.772862    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 02:41:25.785747 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.785708    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 02:41:25.786283 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.786258    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bcf4486-xh5ht"]
Oct 30 02:41:25.887858 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.887822    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-telemeter-client-tls\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.887858 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.887861    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.888108 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.887933    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.888108 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.887978    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-serving-certs-ca-bundle\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.888108 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.888001    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-secret-telemeter-client\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.888108 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.888016    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-federate-client-tls\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.888108 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.888069    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjzn\" (UniqueName: \"kubernetes.io/projected/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-kube-api-access-cmjzn\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.888467 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.888154    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-metrics-client-ca\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.989223 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.989181    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-telemeter-client-tls\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.989223 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.989225    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.989573 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.989270    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.989573 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.989318    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-serving-certs-ca-bundle\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.989573 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.989364    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-secret-telemeter-client\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.989573 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.989387    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-federate-client-tls\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.989573 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.989410    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjzn\" (UniqueName: \"kubernetes.io/projected/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-kube-api-access-cmjzn\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.989790 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.989464    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-metrics-client-ca\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.990185 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.990150    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-serving-certs-ca-bundle\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.990550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.990526    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-metrics-client-ca\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.990894 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.990846    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.992709 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.992628    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-secret-telemeter-client\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.992798 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.992752    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-federate-client-tls\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.993923 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.993897    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:25.994067 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:25.994040    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-telemeter-client-tls\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:26.009714 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:26.009683    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjzn\" (UniqueName: \"kubernetes.io/projected/ebbb58b1-718b-444d-a2ed-bf7cfb7950c2-kube-api-access-cmjzn\") pod \"telemeter-client-bcf4486-xh5ht\" (UID: \"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2\") " pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:26.065340 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:26.065311    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht"
Oct 30 02:41:26.357089 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:26.357047    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5" event={"ID":"5bc64762-491a-420d-aabe-3c8e280bf1ea","Type":"ContainerStarted","Data":"24d4d0c6ee3e6cdc23cc37111210c17474c7d2a4ffb9eddc4d910c0d7ca70b96"}
Oct 30 02:41:26.357600 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:26.357146    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5" event={"ID":"5bc64762-491a-420d-aabe-3c8e280bf1ea","Type":"ContainerStarted","Data":"2f02a31cb17118b49cc6508383d4383ad4fba7b034c404d1a43c1afb12044b6f"}
Oct 30 02:41:26.357600 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:26.357184    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5" event={"ID":"5bc64762-491a-420d-aabe-3c8e280bf1ea","Type":"ContainerStarted","Data":"26babeb1afa75792fec5ac461bae839def5fae649cce004895442a5e1f740cf1"}
Oct 30 02:41:26.357600 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:26.357213    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:26.357600 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:26.357238    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5" event={"ID":"5bc64762-491a-420d-aabe-3c8e280bf1ea","Type":"ContainerStarted","Data":"da7b366c9db2ffe463d5e5c790eaf60066a81bac802827ce0840cac74c299976"}
Oct 30 02:41:26.395100 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:26.395061    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bcf4486-xh5ht"]
Oct 30 02:41:26.404008 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:26.403944    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5" podStartSLOduration=1.8517712579999999 podStartE2EDuration="4.403921054s" totalImagesPullingTime="2.552149796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:26.40297338 +0000 UTC m=+24.327181569" watchObservedRunningTime="2026-10-30 02:41:26.403921054 +0000 UTC m=+24.328129264"
Oct 30 02:41:26.414417 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:26.414382    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbb58b1_718b_444d_a2ed_bf7cfb7950c2.slice/crio-b1fcbb48c46a39bbe37e720ca7647a99e0376316e0df0d976a32e93c6e7d1e8e WatchSource:0}: Error finding container b1fcbb48c46a39bbe37e720ca7647a99e0376316e0df0d976a32e93c6e7d1e8e: Status 404 returned error can't find the container with id b1fcbb48c46a39bbe37e720ca7647a99e0376316e0df0d976a32e93c6e7d1e8e
Oct 30 02:41:27.249544 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.249507    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:41:27.254517 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.254484    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.271804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.271774    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 02:41:27.273242 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.273212    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 02:41:27.273445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.273402    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 02:41:27.273530 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.273450    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 02:41:27.273530 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.273523    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 02:41:27.273626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.273574    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 02:41:27.273626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.273580    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-abc7lc4astsjl\""
Oct 30 02:41:27.273626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.273600    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 02:41:27.275286 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.275266    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kpnzx\""
Oct 30 02:41:27.275457 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.275303    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 02:41:27.275457 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.275331    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 02:41:27.275457 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.275355    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 02:41:27.283761 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.283720    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 02:41:27.283913 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.283793    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 02:41:27.283913 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.283797    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 02:41:27.297017 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.296982    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 02:41:27.299395 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.299364    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 02:41:27.300670 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.300643    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:41:27.302463 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.302402    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lmqlv"
Oct 30 02:41:27.363130 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.363094    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht" event={"ID":"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2","Type":"ContainerStarted","Data":"b1fcbb48c46a39bbe37e720ca7647a99e0376316e0df0d976a32e93c6e7d1e8e"}
Oct 30 02:41:27.406591 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.406552    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.406769 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.406607    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.406769 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.406679    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.406856 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.406798    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.406856 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.406835    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-web-config\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.406947 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.406870    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-config\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.406947 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.406894    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-config-out\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.406947 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.406925    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407062 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.406947    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407062 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.406976    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407062 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.407014    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407062 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.407044    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407182 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.407151    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407216 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.407184    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407246 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.407214    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407286 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.407263    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mczm9\" (UniqueName: \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-kube-api-access-mczm9\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407369 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.407350    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407496 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.407453    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407777 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.407756    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.407855 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.407809    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508304 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508264    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508496 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508320    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508496 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508391    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mczm9\" (UniqueName: \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-kube-api-access-mczm9\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508496 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508451    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508496 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508477    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508779 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508508    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508779 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508532    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508779 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508584    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508779 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508650    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508779 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508684    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508779 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508715    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508779 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508739    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-web-config\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.508779 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508771    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-config\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.509112 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508795    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-config-out\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.509112 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.508842    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.516127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.512042    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.516127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.512116    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.516127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.512340    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.516127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.512385    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.516127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.512541    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.516127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.513723    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.516127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.514939    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.516127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.515946    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.517144 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.516911    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.527335 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.526368    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.527335 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.526923    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.528300 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.528273    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.530148 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.530118    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.530382 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.530337    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.530843 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.530796    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.531500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.531480    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-config-out\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.531776 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.531742    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-config\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.532157 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.532138    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.532368 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.532351    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.532577 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.532557    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-web-config\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.533235 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.533209    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.533797 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.533774    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.533903 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.533812    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.537044 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.536995    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.542661 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.542632    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mczm9\" (UniqueName: \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-kube-api-access-mczm9\") pod \"prometheus-k8s-0\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.586321 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.586283    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:27.921874 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:27.921680    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:41:27.926761 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:27.926726    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf739a539_83fb_43be_ac43_b1e4ae56a686.slice/crio-62d48c9b31c0350f0f57ca31732d45f092812bc737e7098245e7678a737094e7 WatchSource:0}: Error finding container 62d48c9b31c0350f0f57ca31732d45f092812bc737e7098245e7678a737094e7: Status 404 returned error can't find the container with id 62d48c9b31c0350f0f57ca31732d45f092812bc737e7098245e7678a737094e7
Oct 30 02:41:28.368933 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:28.368895    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerStarted","Data":"62d48c9b31c0350f0f57ca31732d45f092812bc737e7098245e7678a737094e7"}
Oct 30 02:41:28.371101 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:28.371068    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht" event={"ID":"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2","Type":"ContainerStarted","Data":"0c0a04669634d1e0b71099bb41abeb76a83906179491b93c64609b5e1f925a02"}
Oct 30 02:41:28.949795 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:28.949756    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f654b9fd-pb7sp"]
Oct 30 02:41:29.377381 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:29.376772    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerStarted","Data":"f78e5f61171b3edefca065369be31d7f5bd00d97ef63b73016f7f6e94e978142"}
Oct 30 02:41:30.063839 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:30.063804    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:30.382258 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:30.382171    4278 generic.go:356] "Generic (PLEG): container finished" podID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerID="f78e5f61171b3edefca065369be31d7f5bd00d97ef63b73016f7f6e94e978142" exitCode=0
Oct 30 02:41:30.382258 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:30.382224    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerDied","Data":"f78e5f61171b3edefca065369be31d7f5bd00d97ef63b73016f7f6e94e978142"}
Oct 30 02:41:30.385215 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:30.385181    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht" event={"ID":"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2","Type":"ContainerStarted","Data":"304fee85e9681bcf4809efc6ed69d8672910847292284cd98d32b97cb1764a45"}
Oct 30 02:41:30.385281 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:30.385250    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht" event={"ID":"ebbb58b1-718b-444d-a2ed-bf7cfb7950c2","Type":"ContainerStarted","Data":"17977f16222271bdd3f51520d1c4c3f88b3456acf57f08e252dc827c7705c6c2"}
Oct 30 02:41:30.483689 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:30.483631    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-bcf4486-xh5ht" podStartSLOduration=2.481751769 podStartE2EDuration="5.483616663s" totalImagesPullingTime="3.001864894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:30.48289028 +0000 UTC m=+28.407098460" watchObservedRunningTime="2026-10-30 02:41:30.483616663 +0000 UTC m=+28.407824852"
Oct 30 02:41:30.590392 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:30.590352    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k684d"
Oct 30 02:41:32.371754 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:32.371720    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7675f749d8-tlnp5"
Oct 30 02:41:32.397491 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:32.397414    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-g6q8h" event={"ID":"9c16167c-eef6-4dc3-85e7-d4a419212e43","Type":"ContainerStarted","Data":"5ae2cfe3b35ca2b9004591ad33099cfb5209790f15d86c88c60f091daa34c00a"}
Oct 30 02:41:32.397856 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:32.397820    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-g6q8h"
Oct 30 02:41:32.411575 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:32.411502    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-g6q8h"
Oct 30 02:41:32.461283 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:32.461221    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-g6q8h" podStartSLOduration=1.570339487 podStartE2EDuration="20.461198486s" totalImagesPullingTime="18.890858999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:32.459225918 +0000 UTC m=+30.383434108" watchObservedRunningTime="2026-10-30 02:41:32.461198486 +0000 UTC m=+30.385406675"
Oct 30 02:41:33.582542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.582508    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-65cd96cfff-dkxwh"]
Oct 30 02:41:33.617502 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.617471    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cd96cfff-dkxwh"]
Oct 30 02:41:33.617682 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.617594    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.677505 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.677470    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-oauth-serving-cert\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.677678 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.677525    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-oauth-config\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.677678 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.677598    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-tmp\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.677678 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.677627    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-service-ca\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.677678 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.677659    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-trusted-ca-bundle\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.677849 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.677765    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rr2\" (UniqueName: \"kubernetes.io/projected/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-kube-api-access-s7rr2\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.677895 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.677846    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-serving-cert\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.677895 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.677886    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-config\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.779038 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.778999    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-serving-cert\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.779232 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.779065    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-config\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.779232 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.779111    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-oauth-serving-cert\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.779232 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.779154    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-oauth-config\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.779232 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.779190    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-tmp\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.779232 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.779221    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-service-ca\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.779538 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.779257    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-trusted-ca-bundle\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.779538 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.779287    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rr2\" (UniqueName: \"kubernetes.io/projected/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-kube-api-access-s7rr2\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.780699 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.779892    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-tmp\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.780699 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.779957    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-oauth-serving-cert\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.780699 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.780050    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-config\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.780699 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.780324    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-service-ca\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.780699 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.780624    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-trusted-ca-bundle\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.782550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.782520    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-serving-cert\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.782650 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.782622    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-oauth-config\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.798365 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.798339    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rr2\" (UniqueName: \"kubernetes.io/projected/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-kube-api-access-s7rr2\") pod \"console-65cd96cfff-dkxwh\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") " pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:33.954864 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:33.954776    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:34.410907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:34.410876    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerStarted","Data":"4a8f0f3daf6c6a6e4d2500e69c78c7bd9b07ad5957a912d21d82b3d839291fc4"}
Oct 30 02:41:34.520766 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:34.520737    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cd96cfff-dkxwh"]
Oct 30 02:41:34.555784 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:41:34.555753    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4766f2_46b3_4956_b4ad_8fa58f9ef8f9.slice/crio-4e861451f19ecc944ea894cba56a23b59dc9ef09fa24dea677dc643f2fcea065 WatchSource:0}: Error finding container 4e861451f19ecc944ea894cba56a23b59dc9ef09fa24dea677dc643f2fcea065: Status 404 returned error can't find the container with id 4e861451f19ecc944ea894cba56a23b59dc9ef09fa24dea677dc643f2fcea065
Oct 30 02:41:35.417761 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:35.417720    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cd96cfff-dkxwh" event={"ID":"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9","Type":"ContainerStarted","Data":"cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147"}
Oct 30 02:41:35.418183 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:35.417824    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cd96cfff-dkxwh" event={"ID":"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9","Type":"ContainerStarted","Data":"4e861451f19ecc944ea894cba56a23b59dc9ef09fa24dea677dc643f2fcea065"}
Oct 30 02:41:35.422225 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:35.422194    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerStarted","Data":"25a5d88eee8c042a2b9f3b954f558e8acf2a5a7e4f052329167e3945df288e4d"}
Oct 30 02:41:35.422349 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:35.422257    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerStarted","Data":"d379329ad6ab03ba2dfb15ccbcc2685812055af9d93665c24c310fbc4bdfe371"}
Oct 30 02:41:35.422349 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:35.422297    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerStarted","Data":"7fff5011a7c93f679230f6cee856968e5fde713da5a99330f297634ef38a9d8b"}
Oct 30 02:41:35.422349 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:35.422323    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerStarted","Data":"8be3868fd049fdedee0a92e248b9a6395673cc21dc91feecf79e2f4f042a7d3c"}
Oct 30 02:41:35.422582 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:35.422363    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerStarted","Data":"b7393f8b6617066da004da639029782ef7948db09635a048376681a523ff9e35"}
Oct 30 02:41:35.458148 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:35.458087    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-65cd96cfff-dkxwh" podStartSLOduration=2.458070491 podStartE2EDuration="2.458070491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:35.457665799 +0000 UTC m=+33.381873986" watchObservedRunningTime="2026-10-30 02:41:35.458070491 +0000 UTC m=+33.382278680"
Oct 30 02:41:35.508247 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:35.508200    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.308816675 podStartE2EDuration="8.508185445s" totalImagesPullingTime="5.174046027s" totalInitContainerRuntime="25.322743ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:35.506231141 +0000 UTC m=+33.430439343" watchObservedRunningTime="2026-10-30 02:41:35.508185445 +0000 UTC m=+33.432393633"
Oct 30 02:41:37.587374 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:37.587341    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:41:43.955230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:43.955181    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:43.955816 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:43.955298    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:43.960415 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:43.960389    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:44.463835 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:44.463805    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:41:44.531921 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:44.531878    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f4d98d8c-r5k8z"]
Oct 30 02:41:53.974240 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:53.974150    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66f654b9fd-pb7sp" podUID="0c6ccb04-6783-4b87-b648-8ab732e4764b" containerName="console" containerID="cri-o://d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985" gracePeriod=15
Oct 30 02:41:54.189058 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:54.189021    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:41:54.249062 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.248991    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:54.368626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.368590    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-oauth-serving-cert\") pod \"0c6ccb04-6783-4b87-b648-8ab732e4764b\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") "
Oct 30 02:41:54.368836 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.368661    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6ccb04-6783-4b87-b648-8ab732e4764b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c6ccb04-6783-4b87-b648-8ab732e4764b-tmp\") pod \"0c6ccb04-6783-4b87-b648-8ab732e4764b\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") "
Oct 30 02:41:54.368836 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.368722    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-oauth-config\") pod \"0c6ccb04-6783-4b87-b648-8ab732e4764b\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") "
Oct 30 02:41:54.368836 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.368757    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-service-ca\") pod \"0c6ccb04-6783-4b87-b648-8ab732e4764b\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") "
Oct 30 02:41:54.368836 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.368777    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-serving-cert\") pod \"0c6ccb04-6783-4b87-b648-8ab732e4764b\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") "
Oct 30 02:41:54.368836 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.368803    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6ccb04-6783-4b87-b648-8ab732e4764b-kube-api-access-84kc9\" (UniqueName: \"kubernetes.io/projected/0c6ccb04-6783-4b87-b648-8ab732e4764b-kube-api-access-84kc9\") pod \"0c6ccb04-6783-4b87-b648-8ab732e4764b\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") "
Oct 30 02:41:54.368836 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.368826    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-config\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-config\") pod \"0c6ccb04-6783-4b87-b648-8ab732e4764b\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") "
Oct 30 02:41:54.369139 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.368855    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-trusted-ca-bundle\") pod \"0c6ccb04-6783-4b87-b648-8ab732e4764b\" (UID: \"0c6ccb04-6783-4b87-b648-8ab732e4764b\") "
Oct 30 02:41:54.369139 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.369001    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6ccb04-6783-4b87-b648-8ab732e4764b-tmp" pod "0c6ccb04-6783-4b87-b648-8ab732e4764b" (UID: "0c6ccb04-6783-4b87-b648-8ab732e4764b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:41:54.369229 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.369144    4278 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c6ccb04-6783-4b87-b648-8ab732e4764b-tmp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:41:54.369229 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.369149    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-oauth-serving-cert" pod "0c6ccb04-6783-4b87-b648-8ab732e4764b" (UID: "0c6ccb04-6783-4b87-b648-8ab732e4764b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:41:54.369333 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.369295    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-service-ca" pod "0c6ccb04-6783-4b87-b648-8ab732e4764b" (UID: "0c6ccb04-6783-4b87-b648-8ab732e4764b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:41:54.369379 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.369338    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-trusted-ca-bundle" pod "0c6ccb04-6783-4b87-b648-8ab732e4764b" (UID: "0c6ccb04-6783-4b87-b648-8ab732e4764b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:41:54.369454 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.369399    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-config" pod "0c6ccb04-6783-4b87-b648-8ab732e4764b" (UID: "0c6ccb04-6783-4b87-b648-8ab732e4764b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:41:54.371112 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.371082    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-serving-cert" pod "0c6ccb04-6783-4b87-b648-8ab732e4764b" (UID: "0c6ccb04-6783-4b87-b648-8ab732e4764b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:41:54.371225 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.371158    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6ccb04-6783-4b87-b648-8ab732e4764b-kube-api-access-84kc9" pod "0c6ccb04-6783-4b87-b648-8ab732e4764b" (UID: "0c6ccb04-6783-4b87-b648-8ab732e4764b"). InnerVolumeSpecName "kube-api-access-84kc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:41:54.371225 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.371165    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-oauth-config" pod "0c6ccb04-6783-4b87-b648-8ab732e4764b" (UID: "0c6ccb04-6783-4b87-b648-8ab732e4764b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:41:54.469696 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.469658    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84kc9\" (UniqueName: \"kubernetes.io/projected/0c6ccb04-6783-4b87-b648-8ab732e4764b-kube-api-access-84kc9\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:41:54.469696 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.469688    4278 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-config\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:41:54.469696 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.469701    4278 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-trusted-ca-bundle\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:41:54.469946 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.469713    4278 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-oauth-serving-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:41:54.469946 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.469725    4278 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-oauth-config\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:41:54.469946 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.469736    4278 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6ccb04-6783-4b87-b648-8ab732e4764b-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:41:54.469946 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.469748    4278 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c6ccb04-6783-4b87-b648-8ab732e4764b-console-serving-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:41:54.497700 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.497662    4278 generic.go:356] "Generic (PLEG): container finished" podID="0c6ccb04-6783-4b87-b648-8ab732e4764b" containerID="d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985" exitCode=0
Oct 30 02:41:54.497887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.497716    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f654b9fd-pb7sp"
Oct 30 02:41:54.497887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.497754    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f654b9fd-pb7sp" event={"ID":"0c6ccb04-6783-4b87-b648-8ab732e4764b","Type":"ContainerDied","Data":"d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985"}
Oct 30 02:41:54.497887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.497816    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f654b9fd-pb7sp" event={"ID":"0c6ccb04-6783-4b87-b648-8ab732e4764b","Type":"ContainerDied","Data":"93683146c33b9ff4bfbba5f29d76149e30bfbd4235e88010e1110384f1ea39dc"}
Oct 30 02:41:54.497887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.497844    4278 scope.go:122] "RemoveContainer" containerID="d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985"
Oct 30 02:41:54.506819 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.506803    4278 scope.go:122] "RemoveContainer" containerID="d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985"
Oct 30 02:41:54.507118 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:41:54.507098    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985\": container with ID starting with d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985 not found: ID does not exist" containerID="d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985"
Oct 30 02:41:54.507165 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.507129    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985"} err="failed to get container status \"d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985\": rpc error: code = NotFound desc = could not find container \"d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985\": container with ID starting with d0ede414b543b7c3699dd05ed69e34057e0d8098a0ea2679b92ff23cc1041985 not found: ID does not exist"
Oct 30 02:41:54.550511 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.550477    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f654b9fd-pb7sp"]
Oct 30 02:41:54.562285 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:54.562252    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66f654b9fd-pb7sp"]
Oct 30 02:41:56.196660 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:41:56.196628    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6ccb04-6783-4b87-b648-8ab732e4764b" path="/var/lib/kubelet/pods/0c6ccb04-6783-4b87-b648-8ab732e4764b/volumes"
Oct 30 02:42:01.237511 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:01.237483    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rt4nr_bf7d6c82-7dde-42d8-b013-120eebeed803/serve-healthcheck-canary/0.log"
Oct 30 02:42:09.552775 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.552697    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f4d98d8c-r5k8z" podUID="32b8af70-8f28-4fae-adf8-c3928acc9fee" containerName="console" containerID="cri-o://09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896" gracePeriod=15
Oct 30 02:42:09.840799 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.840766    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:42:09.898578 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.898546    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-config\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-config\") pod \"32b8af70-8f28-4fae-adf8-c3928acc9fee\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") "
Oct 30 02:42:09.898578 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.898584    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-oauth-config\") pod \"32b8af70-8f28-4fae-adf8-c3928acc9fee\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") "
Oct 30 02:42:09.898828 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.898614    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-service-ca\") pod \"32b8af70-8f28-4fae-adf8-c3928acc9fee\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") "
Oct 30 02:42:09.898828 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.898633    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-serving-cert\") pod \"32b8af70-8f28-4fae-adf8-c3928acc9fee\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") "
Oct 30 02:42:09.898828 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.898652    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b8af70-8f28-4fae-adf8-c3928acc9fee-tmp\" (UniqueName: \"kubernetes.io/empty-dir/32b8af70-8f28-4fae-adf8-c3928acc9fee-tmp\") pod \"32b8af70-8f28-4fae-adf8-c3928acc9fee\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") "
Oct 30 02:42:09.898988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.898837    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b8af70-8f28-4fae-adf8-c3928acc9fee-kube-api-access-tvlfr\" (UniqueName: \"kubernetes.io/projected/32b8af70-8f28-4fae-adf8-c3928acc9fee-kube-api-access-tvlfr\") pod \"32b8af70-8f28-4fae-adf8-c3928acc9fee\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") "
Oct 30 02:42:09.898988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.898949    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-oauth-serving-cert\") pod \"32b8af70-8f28-4fae-adf8-c3928acc9fee\" (UID: \"32b8af70-8f28-4fae-adf8-c3928acc9fee\") "
Oct 30 02:42:09.899092 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.899001    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b8af70-8f28-4fae-adf8-c3928acc9fee-tmp" pod "32b8af70-8f28-4fae-adf8-c3928acc9fee" (UID: "32b8af70-8f28-4fae-adf8-c3928acc9fee"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:09.899092 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.899066    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-service-ca" pod "32b8af70-8f28-4fae-adf8-c3928acc9fee" (UID: "32b8af70-8f28-4fae-adf8-c3928acc9fee"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:09.899203 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.899108    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-config" pod "32b8af70-8f28-4fae-adf8-c3928acc9fee" (UID: "32b8af70-8f28-4fae-adf8-c3928acc9fee"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:09.899356 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.899330    4278 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-config\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:09.899356 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.899355    4278 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:09.899550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.899368    4278 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32b8af70-8f28-4fae-adf8-c3928acc9fee-tmp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:09.899550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.899399    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-oauth-serving-cert" pod "32b8af70-8f28-4fae-adf8-c3928acc9fee" (UID: "32b8af70-8f28-4fae-adf8-c3928acc9fee"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:09.900917 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.900897    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-oauth-config" pod "32b8af70-8f28-4fae-adf8-c3928acc9fee" (UID: "32b8af70-8f28-4fae-adf8-c3928acc9fee"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:09.901289 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.901261    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b8af70-8f28-4fae-adf8-c3928acc9fee-kube-api-access-tvlfr" pod "32b8af70-8f28-4fae-adf8-c3928acc9fee" (UID: "32b8af70-8f28-4fae-adf8-c3928acc9fee"). InnerVolumeSpecName "kube-api-access-tvlfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:09.901289 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:09.901262    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-serving-cert" pod "32b8af70-8f28-4fae-adf8-c3928acc9fee" (UID: "32b8af70-8f28-4fae-adf8-c3928acc9fee"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:10.000222 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.000184    4278 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32b8af70-8f28-4fae-adf8-c3928acc9fee-oauth-serving-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:10.000222 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.000214    4278 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-oauth-config\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:10.000222 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.000223    4278 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32b8af70-8f28-4fae-adf8-c3928acc9fee-console-serving-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:10.000222 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.000232    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvlfr\" (UniqueName: \"kubernetes.io/projected/32b8af70-8f28-4fae-adf8-c3928acc9fee-kube-api-access-tvlfr\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:10.561642 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.561605    4278 generic.go:356] "Generic (PLEG): container finished" podID="32b8af70-8f28-4fae-adf8-c3928acc9fee" containerID="09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896" exitCode=0
Oct 30 02:42:10.562061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.561659    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4d98d8c-r5k8z"
Oct 30 02:42:10.562061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.561664    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4d98d8c-r5k8z" event={"ID":"32b8af70-8f28-4fae-adf8-c3928acc9fee","Type":"ContainerDied","Data":"09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896"}
Oct 30 02:42:10.562061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.561716    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4d98d8c-r5k8z" event={"ID":"32b8af70-8f28-4fae-adf8-c3928acc9fee","Type":"ContainerDied","Data":"641904dc9541d6b353ee3b20e588343dd08908a00789d4349ae0e2400cfbcb47"}
Oct 30 02:42:10.562061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.561736    4278 scope.go:122] "RemoveContainer" containerID="09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896"
Oct 30 02:42:10.569752 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.569733    4278 scope.go:122] "RemoveContainer" containerID="09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896"
Oct 30 02:42:10.570024 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:42:10.570008    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896\": container with ID starting with 09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896 not found: ID does not exist" containerID="09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896"
Oct 30 02:42:10.570072 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.570035    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896"} err="failed to get container status \"09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896\": rpc error: code = NotFound desc = could not find container \"09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896\": container with ID starting with 09231366532a8163ef5ef7be634c9a58c4ae1310fb6ebe286983cf2f1e8d9896 not found: ID does not exist"
Oct 30 02:42:10.610245 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.610217    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f4d98d8c-r5k8z"]
Oct 30 02:42:10.621999 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:10.621970    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f4d98d8c-r5k8z"]
Oct 30 02:42:12.196699 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:12.196667    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b8af70-8f28-4fae-adf8-c3928acc9fee" path="/var/lib/kubelet/pods/32b8af70-8f28-4fae-adf8-c3928acc9fee/volumes"
Oct 30 02:42:24.187758 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:42:24.187721    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:42:33.243242 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:33.243193    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:33.262258 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:33.262226    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:33.677887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:33.677860    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:46.523600 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.523558    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:42:46.524056 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.523898    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="prometheus" containerID="cri-o://4a8f0f3daf6c6a6e4d2500e69c78c7bd9b07ad5957a912d21d82b3d839291fc4" gracePeriod=600
Oct 30 02:42:46.524056 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.523970    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy" containerID="cri-o://d379329ad6ab03ba2dfb15ccbcc2685812055af9d93665c24c310fbc4bdfe371" gracePeriod=600
Oct 30 02:42:46.524056 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.523997    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="thanos-sidecar" containerID="cri-o://8be3868fd049fdedee0a92e248b9a6395673cc21dc91feecf79e2f4f042a7d3c" gracePeriod=600
Oct 30 02:42:46.524056 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.524018    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy-web" containerID="cri-o://7fff5011a7c93f679230f6cee856968e5fde713da5a99330f297634ef38a9d8b" gracePeriod=600
Oct 30 02:42:46.524282 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.524037    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy-thanos" containerID="cri-o://25a5d88eee8c042a2b9f3b954f558e8acf2a5a7e4f052329167e3945df288e4d" gracePeriod=600
Oct 30 02:42:46.524282 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.524063    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="config-reloader" containerID="cri-o://b7393f8b6617066da004da639029782ef7948db09635a048376681a523ff9e35" gracePeriod=600
Oct 30 02:42:46.714993 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.714957    4278 generic.go:356] "Generic (PLEG): container finished" podID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerID="25a5d88eee8c042a2b9f3b954f558e8acf2a5a7e4f052329167e3945df288e4d" exitCode=0
Oct 30 02:42:46.714993 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.714995    4278 generic.go:356] "Generic (PLEG): container finished" podID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerID="d379329ad6ab03ba2dfb15ccbcc2685812055af9d93665c24c310fbc4bdfe371" exitCode=0
Oct 30 02:42:46.715194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.715005    4278 generic.go:356] "Generic (PLEG): container finished" podID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerID="7fff5011a7c93f679230f6cee856968e5fde713da5a99330f297634ef38a9d8b" exitCode=0
Oct 30 02:42:46.715194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.715019    4278 generic.go:356] "Generic (PLEG): container finished" podID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerID="8be3868fd049fdedee0a92e248b9a6395673cc21dc91feecf79e2f4f042a7d3c" exitCode=0
Oct 30 02:42:46.715194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.715028    4278 generic.go:356] "Generic (PLEG): container finished" podID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerID="b7393f8b6617066da004da639029782ef7948db09635a048376681a523ff9e35" exitCode=0
Oct 30 02:42:46.715194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.715035    4278 generic.go:356] "Generic (PLEG): container finished" podID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerID="4a8f0f3daf6c6a6e4d2500e69c78c7bd9b07ad5957a912d21d82b3d839291fc4" exitCode=0
Oct 30 02:42:46.715194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.715032    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerDied","Data":"25a5d88eee8c042a2b9f3b954f558e8acf2a5a7e4f052329167e3945df288e4d"}
Oct 30 02:42:46.715194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.715122    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerDied","Data":"d379329ad6ab03ba2dfb15ccbcc2685812055af9d93665c24c310fbc4bdfe371"}
Oct 30 02:42:46.715194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.715162    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerDied","Data":"7fff5011a7c93f679230f6cee856968e5fde713da5a99330f297634ef38a9d8b"}
Oct 30 02:42:46.715410 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.715199    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerDied","Data":"8be3868fd049fdedee0a92e248b9a6395673cc21dc91feecf79e2f4f042a7d3c"}
Oct 30 02:42:46.715410 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.715236    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerDied","Data":"b7393f8b6617066da004da639029782ef7948db09635a048376681a523ff9e35"}
Oct 30 02:42:46.715410 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.715273    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerDied","Data":"4a8f0f3daf6c6a6e4d2500e69c78c7bd9b07ad5957a912d21d82b3d839291fc4"}
Oct 30 02:42:46.759975 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.759950    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:46.823460 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823359    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-config\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-config\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.823460 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823413    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-0\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.823704 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823461    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-kube-api-access-mczm9\" (UniqueName: \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-kube-api-access-mczm9\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.823704 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823493    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-metrics-client-certs\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.823704 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823524    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-web-config\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-web-config\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.823704 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823669    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.823920 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823730    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-metrics-client-ca\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.823920 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823802    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-tls\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.823920 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823840    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-serving-certs-ca-bundle\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.823920 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823877    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-kube-rbac-proxy\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.823920 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823912    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-trusted-ca-bundle\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.824168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.823962    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-2\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.824168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.824001    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.824168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.824034    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-db\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.824168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.824065    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-tls-assets\" (UniqueName: \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-tls-assets\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.824168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.824098    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-thanos-prometheus-http-client-file\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.824168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.824128    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-kubelet-serving-ca-bundle\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.824582 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.824190    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-grpc-tls\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.824582 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.824222    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-1\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.824582 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.824251    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-config-out\") pod \"f739a539-83fb-43be-ac43-b1e4ae56a686\" (UID: \"f739a539-83fb-43be-ac43-b1e4ae56a686\") "
Oct 30 02:42:46.825784 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.825750    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-kubelet-serving-ca-bundle" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:46.826852 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.826819    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-serving-certs-ca-bundle" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:46.826975 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.826864    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-metrics-client-ca" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:46.827036 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.826970    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-db" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:46.827550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.827380    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-kube-api-access-mczm9" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "kube-api-access-mczm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:46.827550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.827476    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-metrics-client-certs" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:46.827700 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.827646    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-1" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:46.828485 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.828192    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-2" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:46.828485 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.828377    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-trusted-ca-bundle" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:46.828656 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.828487    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-config" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:46.830412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.830387    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-grpc-tls" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:46.830540 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.830514    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-tls-assets" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:46.830712 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.830695    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-thanos-prometheus-http-client-file" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:46.831609 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.831589    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-kube-rbac-proxy" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:46.832672 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.832645    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-0" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:46.832774 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.832721    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-thanos-sidecar-tls" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:46.832774 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.832726    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-config-out" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:46.832774 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.832755    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-kube-rbac-proxy-web" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:46.833993 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.833966    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-tls" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:46.839810 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.839788    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-web-config" pod "f739a539-83fb-43be-ac43-b1e4ae56a686" (UID: "f739a539-83fb-43be-ac43-b1e4ae56a686"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:46.925140 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925101    4278 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-grpc-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925140 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925134    4278 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925140 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925143    4278 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-config-out\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925140 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925153    4278 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-config\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925162    4278 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925171    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mczm9\" (UniqueName: \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-kube-api-access-mczm9\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925180    4278 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-metrics-client-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925189    4278 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-web-config\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925199    4278 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925209    4278 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-metrics-client-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925217    4278 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925225    4278 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925237    4278 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-kube-rbac-proxy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925246    4278 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925256    4278 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925265    4278 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925273    4278 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f739a539-83fb-43be-ac43-b1e4ae56a686-prometheus-k8s-db\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925281    4278 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f739a539-83fb-43be-ac43-b1e4ae56a686-tls-assets\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925289    4278 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f739a539-83fb-43be-ac43-b1e4ae56a686-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:46.925412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:46.925297    4278 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f739a539-83fb-43be-ac43-b1e4ae56a686-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:42:47.197990 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.197894    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8454899578-4xqff"]
Oct 30 02:42:47.198318 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198294    4278 cpu_manager.go:414] "containerMap: removing container" podUID="32b8af70-8f28-4fae-adf8-c3928acc9fee" containerName="console"
Oct 30 02:42:47.198318 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198317    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b8af70-8f28-4fae-adf8-c3928acc9fee" containerName="console"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198331    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="thanos-sidecar"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198338    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="thanos-sidecar"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198368    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="init-config-reloader"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198377    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="init-config-reloader"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198396    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="config-reloader"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198403    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="config-reloader"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198417    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0c6ccb04-6783-4b87-b648-8ab732e4764b" containerName="console"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198439    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6ccb04-6783-4b87-b648-8ab732e4764b" containerName="console"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198456    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy-thanos"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198463    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy-thanos"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198476    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198483    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198496    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="prometheus"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198503    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="prometheus"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198513    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy-web"
Oct 30 02:42:47.198532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198520    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy-web"
Oct 30 02:42:47.199306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198599    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="config-reloader"
Oct 30 02:42:47.199306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198613    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6ccb04-6783-4b87-b648-8ab732e4764b" containerName="console"
Oct 30 02:42:47.199306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198623    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy-thanos"
Oct 30 02:42:47.199306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198634    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy"
Oct 30 02:42:47.199306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198642    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="prometheus"
Oct 30 02:42:47.199306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198651    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="kube-rbac-proxy-web"
Oct 30 02:42:47.199306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198659    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b8af70-8f28-4fae-adf8-c3928acc9fee" containerName="console"
Oct 30 02:42:47.199306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.198667    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" containerName="thanos-sidecar"
Oct 30 02:42:47.201553 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.201531    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.223844 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.223812    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8454899578-4xqff"]
Oct 30 02:42:47.228010 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.227983    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ad111a8-76cf-488a-821a-8c456abc141c-console-oauth-config\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.228141 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.228041    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8blg\" (UniqueName: \"kubernetes.io/projected/3ad111a8-76cf-488a-821a-8c456abc141c-kube-api-access-n8blg\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.228198 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.228150    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ad111a8-76cf-488a-821a-8c456abc141c-console-serving-cert\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.228198 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.228194    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ad111a8-76cf-488a-821a-8c456abc141c-tmp\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.228286 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.228212    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-console-config\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.228286 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.228247    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-service-ca\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.228378 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.228290    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-oauth-serving-cert\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.228378 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.228347    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-trusted-ca-bundle\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.329684 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.329644    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-service-ca\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.329684 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.329681    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-oauth-serving-cert\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.329899 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.329698    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-trusted-ca-bundle\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.329899 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.329819    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ad111a8-76cf-488a-821a-8c456abc141c-console-oauth-config\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.329961 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.329901    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8blg\" (UniqueName: \"kubernetes.io/projected/3ad111a8-76cf-488a-821a-8c456abc141c-kube-api-access-n8blg\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.329961 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.329950    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ad111a8-76cf-488a-821a-8c456abc141c-console-serving-cert\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.330039 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.330001    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ad111a8-76cf-488a-821a-8c456abc141c-tmp\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.330039 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.330030    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-console-config\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.330390 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.330356    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ad111a8-76cf-488a-821a-8c456abc141c-tmp\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.330549 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.330501    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-service-ca\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.330746 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.330723    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-console-config\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.330846 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.330750    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-trusted-ca-bundle\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.330846 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.330771    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ad111a8-76cf-488a-821a-8c456abc141c-oauth-serving-cert\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.332379 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.332353    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ad111a8-76cf-488a-821a-8c456abc141c-console-oauth-config\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.332578 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.332558    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ad111a8-76cf-488a-821a-8c456abc141c-console-serving-cert\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.349362 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.349337    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8blg\" (UniqueName: \"kubernetes.io/projected/3ad111a8-76cf-488a-821a-8c456abc141c-kube-api-access-n8blg\") pod \"console-8454899578-4xqff\" (UID: \"3ad111a8-76cf-488a-821a-8c456abc141c\") " pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.511873 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.511833    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:47.721246 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.721213    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f739a539-83fb-43be-ac43-b1e4ae56a686","Type":"ContainerDied","Data":"62d48c9b31c0350f0f57ca31732d45f092812bc737e7098245e7678a737094e7"}
Oct 30 02:42:47.721724 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.721289    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.721724 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.721319    4278 scope.go:122] "RemoveContainer" containerID="25a5d88eee8c042a2b9f3b954f558e8acf2a5a7e4f052329167e3945df288e4d"
Oct 30 02:42:47.729181 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.729165    4278 scope.go:122] "RemoveContainer" containerID="d379329ad6ab03ba2dfb15ccbcc2685812055af9d93665c24c310fbc4bdfe371"
Oct 30 02:42:47.736175 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.736150    4278 scope.go:122] "RemoveContainer" containerID="7fff5011a7c93f679230f6cee856968e5fde713da5a99330f297634ef38a9d8b"
Oct 30 02:42:47.742842 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.742823    4278 scope.go:122] "RemoveContainer" containerID="8be3868fd049fdedee0a92e248b9a6395673cc21dc91feecf79e2f4f042a7d3c"
Oct 30 02:42:47.749560 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.749539    4278 scope.go:122] "RemoveContainer" containerID="b7393f8b6617066da004da639029782ef7948db09635a048376681a523ff9e35"
Oct 30 02:42:47.756219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.756200    4278 scope.go:122] "RemoveContainer" containerID="4a8f0f3daf6c6a6e4d2500e69c78c7bd9b07ad5957a912d21d82b3d839291fc4"
Oct 30 02:42:47.763690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.763664    4278 scope.go:122] "RemoveContainer" containerID="f78e5f61171b3edefca065369be31d7f5bd00d97ef63b73016f7f6e94e978142"
Oct 30 02:42:47.773529 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.773507    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:42:47.785277 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.785253    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:42:47.808389 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.808361    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:42:47.812263 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.812236    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.827417 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827392    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 02:42:47.827682 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827581    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 02:42:47.827682 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827588    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 02:42:47.827682 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827631    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 02:42:47.827682 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827642    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 02:42:47.827682 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827649    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 02:42:47.827682 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827654    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 02:42:47.827682 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827631    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kpnzx\""
Oct 30 02:42:47.828023 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827713    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 02:42:47.828023 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827761    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-abc7lc4astsjl\""
Oct 30 02:42:47.828023 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827788    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 02:42:47.828023 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827811    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 02:42:47.828023 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.827837    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 02:42:47.828335 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.828082    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 02:42:47.828335 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.828147    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 02:42:47.834556 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.834532    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.834675 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.834579    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-config\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.834738 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.834667    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.834815 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.834793    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.834868 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.834833    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.834922 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.834901    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.834971 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.834941    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835018 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.834972    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835018 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835004    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835119 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835083    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835173 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835116    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835173 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835147    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835237 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835192    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8fm\" (UniqueName: \"kubernetes.io/projected/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-kube-api-access-4p8fm\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835269 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835239    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835268    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835349 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835320    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835385 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835357    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835438 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835381    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835438 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835403    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.835516 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.835442    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.838538 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:42:47.838515    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad111a8_76cf_488a_821a_8c456abc141c.slice/crio-d7766bc3fc4fb265b16d90ccb99a426aff76721a8dc1dc18cf2980e3fa3745d0 WatchSource:0}: Error finding container d7766bc3fc4fb265b16d90ccb99a426aff76721a8dc1dc18cf2980e3fa3745d0: Status 404 returned error can't find the container with id d7766bc3fc4fb265b16d90ccb99a426aff76721a8dc1dc18cf2980e3fa3745d0
Oct 30 02:42:47.841061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.841041    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 02:42:47.841442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.841407    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 02:42:47.842820 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.842803    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8454899578-4xqff"]
Oct 30 02:42:47.846320 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.846293    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:42:47.936041 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936004    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936041 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936041    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936063    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936082    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936102    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936240    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936279    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936322    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-config\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936355    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936387    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936415    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936459    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936481    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936503    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936524    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936614    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936653    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936682    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936731    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8fm\" (UniqueName: \"kubernetes.io/projected/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-kube-api-access-4p8fm\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.936804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936802    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.937787 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.936876    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.937787 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.937348    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.937787 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.937491    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.938348 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.938318    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.938955 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.938929    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.939368 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.939109    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.939542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.939512    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.940337 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.940309    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.941318 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.941290    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.942164 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.941785    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.942164 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.941795    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.942164 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.942103    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.942164 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.942124    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.943191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.943163    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.943292 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.943244    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.943458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.943413    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.943605 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.943587    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.943826 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.943807    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.943936 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.943917    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-config\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:47.958319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:47.958282    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8fm\" (UniqueName: \"kubernetes.io/projected/9f7aea3a-e14a-4eec-8d24-41d4d56a8c84-kube-api-access-4p8fm\") pod \"prometheus-k8s-0\" (UID: \"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:48.130052 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:48.129953    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:48.197656 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:48.197620    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f739a539-83fb-43be-ac43-b1e4ae56a686" path="/var/lib/kubelet/pods/f739a539-83fb-43be-ac43-b1e4ae56a686/volumes"
Oct 30 02:42:48.455127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:48.455031    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:42:48.459643 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:42:48.459617    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7aea3a_e14a_4eec_8d24_41d4d56a8c84.slice/crio-cf13cb7701cbf8b439f8bbafa86cb445d6d37f1246b6ddc788e15554fdabca88 WatchSource:0}: Error finding container cf13cb7701cbf8b439f8bbafa86cb445d6d37f1246b6ddc788e15554fdabca88: Status 404 returned error can't find the container with id cf13cb7701cbf8b439f8bbafa86cb445d6d37f1246b6ddc788e15554fdabca88
Oct 30 02:42:48.726735 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:48.726627    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8454899578-4xqff" event={"ID":"3ad111a8-76cf-488a-821a-8c456abc141c","Type":"ContainerStarted","Data":"1c0f43fae50adf342e2634be36a0eb483ce7f4e74fa43cbbf278a6e5924b8d8f"}
Oct 30 02:42:48.726735 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:48.726686    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8454899578-4xqff" event={"ID":"3ad111a8-76cf-488a-821a-8c456abc141c","Type":"ContainerStarted","Data":"d7766bc3fc4fb265b16d90ccb99a426aff76721a8dc1dc18cf2980e3fa3745d0"}
Oct 30 02:42:48.729239 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:48.729214    4278 generic.go:356] "Generic (PLEG): container finished" podID="9f7aea3a-e14a-4eec-8d24-41d4d56a8c84" containerID="4bfc52cb82e108f05f44aa508d45091f3bcdd7248ddea353a63c05750f803a21" exitCode=0
Oct 30 02:42:48.733696 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:48.729618    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84","Type":"ContainerDied","Data":"4bfc52cb82e108f05f44aa508d45091f3bcdd7248ddea353a63c05750f803a21"}
Oct 30 02:42:48.733696 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:48.730515    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84","Type":"ContainerStarted","Data":"cf13cb7701cbf8b439f8bbafa86cb445d6d37f1246b6ddc788e15554fdabca88"}
Oct 30 02:42:48.766045 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:48.765996    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8454899578-4xqff" podStartSLOduration=1.7659804129999999 podStartE2EDuration="1.765980413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:48.765615769 +0000 UTC m=+106.689823956" watchObservedRunningTime="2026-10-30 02:42:48.765980413 +0000 UTC m=+106.690188601"
Oct 30 02:42:49.736025 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:49.735988    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84","Type":"ContainerStarted","Data":"602db9a21247f68237a190101853454b37d8b15dc62e92e9e96fea7cdbc0166b"}
Oct 30 02:42:49.736536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:49.736072    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84","Type":"ContainerStarted","Data":"44af0a872bf1dff12b69f95bd0cffb10b0ed282b35ad6e6c7bc895f856f433bc"}
Oct 30 02:42:49.736536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:49.736115    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84","Type":"ContainerStarted","Data":"1d27d2147c45211f87d79032e45740e72fee407e695ec0e6741fd6ea5b7c6069"}
Oct 30 02:42:49.736536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:49.736197    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84","Type":"ContainerStarted","Data":"beaf2a34e04499557b231e26b921c07a38b0eb85b3023f439ba4fe450370620f"}
Oct 30 02:42:49.736536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:49.736245    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84","Type":"ContainerStarted","Data":"00e7fa8df8e4c9ad5ed79f2dae52cce1075de54da75991eea30b1e2742996f7b"}
Oct 30 02:42:49.736536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:49.736284    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f7aea3a-e14a-4eec-8d24-41d4d56a8c84","Type":"ContainerStarted","Data":"8a1829a66e43898d93239d17545408135d809dcfab49b057bdf479a344c910f0"}
Oct 30 02:42:49.798719 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:49.798664    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7794439459999998 podStartE2EDuration="2.798644393s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.200447ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:49.797559917 +0000 UTC m=+107.721768104" watchObservedRunningTime="2026-10-30 02:42:49.798644393 +0000 UTC m=+107.722852584"
Oct 30 02:42:53.130122 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:53.130082    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:42:54.188493 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:42:54.188454    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:42:57.512238 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:57.512185    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:57.512238 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:57.512250    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:57.517189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:57.517162    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:57.776005 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:57.775934    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8454899578-4xqff"
Oct 30 02:42:57.840636 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:42:57.840601    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65cd96cfff-dkxwh"]
Oct 30 02:43:22.860927 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:22.860876    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-65cd96cfff-dkxwh" podUID="bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" containerName="console" containerID="cri-o://cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147" gracePeriod=15
Oct 30 02:43:23.103082 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.103055    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:43:23.147721 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.147629    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-trusted-ca-bundle\") pod \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") "
Oct 30 02:43:23.147721 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.147699    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-tmp\") pod \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") "
Oct 30 02:43:23.147963 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.147738    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-oauth-serving-cert\") pod \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") "
Oct 30 02:43:23.147963 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.147767    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-service-ca\") pod \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") "
Oct 30 02:43:23.147963 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.147793    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-kube-api-access-s7rr2\" (UniqueName: \"kubernetes.io/projected/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-kube-api-access-s7rr2\") pod \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") "
Oct 30 02:43:23.147963 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.147822    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-oauth-config\") pod \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") "
Oct 30 02:43:23.147963 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.147851    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-serving-cert\") pod \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") "
Oct 30 02:43:23.147963 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.147878    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-config\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-config\") pod \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\" (UID: \"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9\") "
Oct 30 02:43:23.148256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.148006    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-tmp" pod "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" (UID: "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:43:23.148256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.148149    4278 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-tmp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:43:23.148256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.148225    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-trusted-ca-bundle" pod "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" (UID: "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:23.148385 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.148259    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-service-ca" pod "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" (UID: "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:23.148385 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.148267    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-oauth-serving-cert" pod "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" (UID: "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:23.148385 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.148319    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-config" pod "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" (UID: "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:23.150150 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.150114    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-oauth-config" pod "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" (UID: "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:43:23.150327 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.150220    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-kube-api-access-s7rr2" pod "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" (UID: "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9"). InnerVolumeSpecName "kube-api-access-s7rr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:43:23.150857 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.150821    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-serving-cert" pod "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" (UID: "bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:43:23.248821 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.248779    4278 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-serving-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:43:23.248821 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.248814    4278 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-config\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:43:23.248821 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.248824    4278 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-trusted-ca-bundle\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:43:23.248821 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.248832    4278 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-oauth-serving-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:43:23.249097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.248841    4278 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:43:23.249097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.248849    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rr2\" (UniqueName: \"kubernetes.io/projected/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-kube-api-access-s7rr2\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:43:23.249097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.248859    4278 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9-console-oauth-config\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:43:23.873470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.873436    4278 generic.go:356] "Generic (PLEG): container finished" podID="bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" containerID="cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147" exitCode=0
Oct 30 02:43:23.873887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.873484    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cd96cfff-dkxwh"
Oct 30 02:43:23.873887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.873486    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cd96cfff-dkxwh" event={"ID":"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9","Type":"ContainerDied","Data":"cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147"}
Oct 30 02:43:23.873887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.873623    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cd96cfff-dkxwh" event={"ID":"bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9","Type":"ContainerDied","Data":"4e861451f19ecc944ea894cba56a23b59dc9ef09fa24dea677dc643f2fcea065"}
Oct 30 02:43:23.873887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.873654    4278 scope.go:122] "RemoveContainer" containerID="cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147"
Oct 30 02:43:23.881902 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.881878    4278 scope.go:122] "RemoveContainer" containerID="cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147"
Oct 30 02:43:23.882159 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:43:23.882141    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147\": container with ID starting with cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147 not found: ID does not exist" containerID="cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147"
Oct 30 02:43:23.882209 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.882171    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147"} err="failed to get container status \"cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147\": rpc error: code = NotFound desc = could not find container \"cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147\": container with ID starting with cf562bbf7e32ee0dddede05794c8434b0b2a3da78d5a205408c97e49ba300147 not found: ID does not exist"
Oct 30 02:43:23.928587 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.924976    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65cd96cfff-dkxwh"]
Oct 30 02:43:23.936078 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:23.936046    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65cd96cfff-dkxwh"]
Oct 30 02:43:24.188551 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:43:24.188465    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:43:24.196087 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:24.196061    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" path="/var/lib/kubelet/pods/bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9/volumes"
Oct 30 02:43:48.131145 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:48.131088    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:43:48.146942 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:48.146917    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:43:48.980278 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:43:48.980244    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:43:54.189325 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:43:54.189287    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:44:24.189115 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:44:24.189063    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:44:54.189032 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:44:54.188950    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:45:24.188976 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:45:24.188934    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:45:35.101532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.101497    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"]
Oct 30 02:45:35.102008 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.101818    4278 cpu_manager.go:414] "containerMap: removing container" podUID="bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" containerName="console"
Oct 30 02:45:35.102008 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.101828    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" containerName="console"
Oct 30 02:45:35.102008 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.101874    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4766f2-46b3-4956-b4ad-8fa58f9ef8f9" containerName="console"
Oct 30 02:45:35.104783 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.104765    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"
Oct 30 02:45:35.135497 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.135466    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 02:45:35.135497 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.135465    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:45:35.135685 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.135501    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5bl6v\""
Oct 30 02:45:35.136632 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.136615    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"]
Oct 30 02:45:35.159496 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.159466    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gc7t\" (UniqueName: \"kubernetes.io/projected/3db8fb46-2bfc-4755-a334-b00de32e67f0-kube-api-access-4gc7t\") pod \"openshift-lws-operator-8465994787-hrg5d\" (UID: \"3db8fb46-2bfc-4755-a334-b00de32e67f0\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"
Oct 30 02:45:35.159710 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.159585    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3db8fb46-2bfc-4755-a334-b00de32e67f0-tmp\") pod \"openshift-lws-operator-8465994787-hrg5d\" (UID: \"3db8fb46-2bfc-4755-a334-b00de32e67f0\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"
Oct 30 02:45:35.260908 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.260869    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3db8fb46-2bfc-4755-a334-b00de32e67f0-tmp\") pod \"openshift-lws-operator-8465994787-hrg5d\" (UID: \"3db8fb46-2bfc-4755-a334-b00de32e67f0\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"
Oct 30 02:45:35.261088 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.260921    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gc7t\" (UniqueName: \"kubernetes.io/projected/3db8fb46-2bfc-4755-a334-b00de32e67f0-kube-api-access-4gc7t\") pod \"openshift-lws-operator-8465994787-hrg5d\" (UID: \"3db8fb46-2bfc-4755-a334-b00de32e67f0\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"
Oct 30 02:45:35.261255 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.261235    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3db8fb46-2bfc-4755-a334-b00de32e67f0-tmp\") pod \"openshift-lws-operator-8465994787-hrg5d\" (UID: \"3db8fb46-2bfc-4755-a334-b00de32e67f0\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"
Oct 30 02:45:35.282029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.282003    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gc7t\" (UniqueName: \"kubernetes.io/projected/3db8fb46-2bfc-4755-a334-b00de32e67f0-kube-api-access-4gc7t\") pod \"openshift-lws-operator-8465994787-hrg5d\" (UID: \"3db8fb46-2bfc-4755-a334-b00de32e67f0\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"
Oct 30 02:45:35.431162 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.431087    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"
Oct 30 02:45:35.725123 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:35.724964    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d"]
Oct 30 02:45:35.727918 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:45:35.727891    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3db8fb46_2bfc_4755_a334_b00de32e67f0.slice/crio-6adcb7fca8ad46bf644ac6a81c16d0ca4a8e3a9fe5d8bc881eb9099c414198ae WatchSource:0}: Error finding container 6adcb7fca8ad46bf644ac6a81c16d0ca4a8e3a9fe5d8bc881eb9099c414198ae: Status 404 returned error can't find the container with id 6adcb7fca8ad46bf644ac6a81c16d0ca4a8e3a9fe5d8bc881eb9099c414198ae
Oct 30 02:45:36.340145 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:36.340109    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d" event={"ID":"3db8fb46-2bfc-4755-a334-b00de32e67f0","Type":"ContainerStarted","Data":"6adcb7fca8ad46bf644ac6a81c16d0ca4a8e3a9fe5d8bc881eb9099c414198ae"}
Oct 30 02:45:38.355119 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:38.355068    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d" event={"ID":"3db8fb46-2bfc-4755-a334-b00de32e67f0","Type":"ContainerStarted","Data":"c237b675f4264f20d3062dae86bdb0f05b664998bf3724e91fcb098ff183544d"}
Oct 30 02:45:38.394794 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:38.394720    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-hrg5d" podStartSLOduration=0.840426113 podStartE2EDuration="3.394701009s" totalImagesPullingTime="2.554274896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:38.394159822 +0000 UTC m=+276.318368014" watchObservedRunningTime="2026-10-30 02:45:38.394701009 +0000 UTC m=+276.318909227"
Oct 30 02:45:52.485745 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.485711    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"]
Oct 30 02:45:52.492496 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.492468    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.511242 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.511210    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"]
Oct 30 02:45:52.569075 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.569038    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 02:45:52.569075 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.569049    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 02:45:52.569298 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.569122    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 02:45:52.569298 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.569136    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-knctd\""
Oct 30 02:45:52.569298 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.569263    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 02:45:52.569298 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.569278    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 02:45:52.608722 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.608683    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-metrics-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.608890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.608728    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-manifests\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.608890 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.608836    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.608970 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.608915    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-tmp\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.608970 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.608956    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwp9q\" (UniqueName: \"kubernetes.io/projected/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-kube-api-access-gwp9q\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.609047 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.608976    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-webhook-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.709476 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.709408    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-metrics-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.709476 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.709479    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-manifests\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.709715 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.709526    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.709715 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.709552    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-tmp\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.709715 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.709606    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwp9q\" (UniqueName: \"kubernetes.io/projected/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-kube-api-access-gwp9q\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.709715 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.709634    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-webhook-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.710159 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.710124    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-manifests\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.712666 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.712615    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-tmp\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.712920 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.712898    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-webhook-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.713315 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.713292    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.713459 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.713439    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-metrics-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.733577 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.733550    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwp9q\" (UniqueName: \"kubernetes.io/projected/1c53f0f1-2c25-41cc-8049-a17305d2dc2c-kube-api-access-gwp9q\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b7ftg\" (UID: \"1c53f0f1-2c25-41cc-8049-a17305d2dc2c\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:52.808584 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:52.808538    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:53.228403 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:53.228374    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"]
Oct 30 02:45:53.231069 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:45:53.231044    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c53f0f1_2c25_41cc_8049_a17305d2dc2c.slice/crio-749c872de08ef57e2cc33ea1b2465c1437b6479a9c3f634635456baa49487498 WatchSource:0}: Error finding container 749c872de08ef57e2cc33ea1b2465c1437b6479a9c3f634635456baa49487498: Status 404 returned error can't find the container with id 749c872de08ef57e2cc33ea1b2465c1437b6479a9c3f634635456baa49487498
Oct 30 02:45:53.412371 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:53.412333    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg" event={"ID":"1c53f0f1-2c25-41cc-8049-a17305d2dc2c","Type":"ContainerStarted","Data":"749c872de08ef57e2cc33ea1b2465c1437b6479a9c3f634635456baa49487498"}
Oct 30 02:45:54.196035 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:45:54.195999    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:45:56.427548 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:56.427453    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg" event={"ID":"1c53f0f1-2c25-41cc-8049-a17305d2dc2c","Type":"ContainerStarted","Data":"2650a777f37fa3c215d03e11c68ad439c9ee60933e0e92ad53fe31329502e425"}
Oct 30 02:45:57.432887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:57.432855    4278 generic.go:356] "Generic (PLEG): container finished" podID="1c53f0f1-2c25-41cc-8049-a17305d2dc2c" containerID="2650a777f37fa3c215d03e11c68ad439c9ee60933e0e92ad53fe31329502e425" exitCode=0
Oct 30 02:45:57.433313 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:57.432898    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg" event={"ID":"1c53f0f1-2c25-41cc-8049-a17305d2dc2c","Type":"ContainerDied","Data":"2650a777f37fa3c215d03e11c68ad439c9ee60933e0e92ad53fe31329502e425"}
Oct 30 02:45:58.438804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:58.438766    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg" event={"ID":"1c53f0f1-2c25-41cc-8049-a17305d2dc2c","Type":"ContainerStarted","Data":"1d3ddbabb9b9834e9de350d488f3e8aeea08db9ada25e0d183b11369f631e28f"}
Oct 30 02:45:58.439259 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:58.438910    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:45:58.522484 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:45:58.522409    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg" podStartSLOduration=2.840991663 podStartE2EDuration="6.522394242s" totalImagesPullingTime="3.198033252s" totalInitContainerRuntime="483.369327ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:58.507989057 +0000 UTC m=+296.432197279" watchObservedRunningTime="2026-10-30 02:45:58.522394242 +0000 UTC m=+296.446602430"
Oct 30 02:46:09.445796 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:09.445765    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b7ftg"
Oct 30 02:46:14.668007 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.667972    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"]
Oct 30 02:46:14.675596 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.675567    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.691655 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.691631    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"]
Oct 30 02:46:14.703761 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.703697    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 02:46:14.703761 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.703734    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 02:46:14.704978 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.704962    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 02:46:14.705027 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.705001    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kqx47\""
Oct 30 02:46:14.797706 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.797672    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-cert\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.797706 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.797706    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-metrics-cert\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.797925 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.797778    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-manager-config\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.797925 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.797867    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hb6\" (UniqueName: \"kubernetes.io/projected/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-kube-api-access-j4hb6\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.898229 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.898189    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-manager-config\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.898458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.898256    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hb6\" (UniqueName: \"kubernetes.io/projected/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-kube-api-access-j4hb6\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.898458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.898312    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-cert\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.898458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.898338    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-metrics-cert\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.898913 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.898888    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-manager-config\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.901038 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.901015    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-metrics-cert\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.901147 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.901092    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-cert\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.918070 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.918018    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hb6\" (UniqueName: \"kubernetes.io/projected/e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a-kube-api-access-j4hb6\") pod \"lws-controller-manager-5548b4dc5c-5vbrn\" (UID: \"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:14.987588 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:14.987556    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:15.278470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.278099    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"]
Oct 30 02:46:15.281897 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:46:15.281864    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d0910b_7e3f_4ddf_aa6d_5a1d7cb2f27a.slice/crio-d235a9c77825d778c5994f5d1c45dfffd843b64fb0dfa550f882bf37f30f00fa WatchSource:0}: Error finding container d235a9c77825d778c5994f5d1c45dfffd843b64fb0dfa550f882bf37f30f00fa: Status 404 returned error can't find the container with id d235a9c77825d778c5994f5d1c45dfffd843b64fb0dfa550f882bf37f30f00fa
Oct 30 02:46:15.283812 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.283798    4278 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:46:15.374941 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.374909    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-95rk5"]
Oct 30 02:46:15.378311 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.378295    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:15.396584 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.396558    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-nnvrz\""
Oct 30 02:46:15.396584 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.396563    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 30 02:46:15.398135 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.398111    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-95rk5"]
Oct 30 02:46:15.402061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.402035    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcp7\" (UniqueName: \"kubernetes.io/projected/429df927-d84d-4d8c-9111-263b24400651-kube-api-access-8lcp7\") pod \"odh-observability-584f99c788-95rk5\" (UID: \"429df927-d84d-4d8c-9111-263b24400651\") " pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:15.402152 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.402092    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/429df927-d84d-4d8c-9111-263b24400651-webhook-certs\") pod \"odh-observability-584f99c788-95rk5\" (UID: \"429df927-d84d-4d8c-9111-263b24400651\") " pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:15.503177 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.503144    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcp7\" (UniqueName: \"kubernetes.io/projected/429df927-d84d-4d8c-9111-263b24400651-kube-api-access-8lcp7\") pod \"odh-observability-584f99c788-95rk5\" (UID: \"429df927-d84d-4d8c-9111-263b24400651\") " pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:15.503385 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.503194    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/429df927-d84d-4d8c-9111-263b24400651-webhook-certs\") pod \"odh-observability-584f99c788-95rk5\" (UID: \"429df927-d84d-4d8c-9111-263b24400651\") " pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:15.503385 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:46:15.503339    4278 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 02:46:15.503537 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:46:15.503464    4278 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429df927-d84d-4d8c-9111-263b24400651-webhook-certs podName:429df927-d84d-4d8c-9111-263b24400651 nodeName:}" failed. No retries permitted until 2026-10-30 02:46:16.003412566 +0000 UTC m=+313.927620745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/429df927-d84d-4d8c-9111-263b24400651-webhook-certs") pod "odh-observability-584f99c788-95rk5" (UID: "429df927-d84d-4d8c-9111-263b24400651") : secret "odh-observability-webhook-cert" not found
Oct 30 02:46:15.506637 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.506609    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn" event={"ID":"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a","Type":"ContainerStarted","Data":"d235a9c77825d778c5994f5d1c45dfffd843b64fb0dfa550f882bf37f30f00fa"}
Oct 30 02:46:15.523608 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:15.523573    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcp7\" (UniqueName: \"kubernetes.io/projected/429df927-d84d-4d8c-9111-263b24400651-kube-api-access-8lcp7\") pod \"odh-observability-584f99c788-95rk5\" (UID: \"429df927-d84d-4d8c-9111-263b24400651\") " pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:16.007497 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:16.007455    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/429df927-d84d-4d8c-9111-263b24400651-webhook-certs\") pod \"odh-observability-584f99c788-95rk5\" (UID: \"429df927-d84d-4d8c-9111-263b24400651\") " pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:16.010786 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:16.010753    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/429df927-d84d-4d8c-9111-263b24400651-webhook-certs\") pod \"odh-observability-584f99c788-95rk5\" (UID: \"429df927-d84d-4d8c-9111-263b24400651\") " pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:16.291303 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:16.291223    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:16.607401 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:16.607299    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-95rk5"]
Oct 30 02:46:16.611661 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:46:16.611628    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429df927_d84d_4d8c_9111_263b24400651.slice/crio-c77a45b342c049b2bc7875ed8b55e35c90766b2787064934ef570ec830042790 WatchSource:0}: Error finding container c77a45b342c049b2bc7875ed8b55e35c90766b2787064934ef570ec830042790: Status 404 returned error can't find the container with id c77a45b342c049b2bc7875ed8b55e35c90766b2787064934ef570ec830042790
Oct 30 02:46:17.522317 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.522269    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn" event={"ID":"e2d0910b-7e3f-4ddf-aa6d-5a1d7cb2f27a","Type":"ContainerStarted","Data":"fbd6994283fe0bf775a1cbbf4f7b799c83db9f87aca053a16a54a64aa3e1da78"}
Oct 30 02:46:17.522741 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.522472    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:17.523907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.523868    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-95rk5" event={"ID":"429df927-d84d-4d8c-9111-263b24400651","Type":"ContainerStarted","Data":"c77a45b342c049b2bc7875ed8b55e35c90766b2787064934ef570ec830042790"}
Oct 30 02:46:17.567152 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.567096    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn" podStartSLOduration=1.900010345 podStartE2EDuration="3.567079513s" totalImagesPullingTime="1.667069168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:46:17.565166562 +0000 UTC m=+315.489374777" watchObservedRunningTime="2026-10-30 02:46:17.567079513 +0000 UTC m=+315.491287781"
Oct 30 02:46:17.955715 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.955639    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"]
Oct 30 02:46:17.962975 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.962942    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:17.984834 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.984806    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 02:46:17.985683 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.985659    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 02:46:17.985828 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.985774    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 02:46:17.997398 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.997360    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-j95qq\""
Oct 30 02:46:17.999006 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:17.998973    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"]
Oct 30 02:46:18.026008 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.025965    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.026168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.026019    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.026168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.026097    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.026168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.026151    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.026322 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.026176    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.026322 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.026258    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.026322 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.026288    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.026490 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.026352    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mnr\" (UniqueName: \"kubernetes.io/projected/f1ac3110-5846-48f8-a0d4-edf87b4defb1-kube-api-access-x9mnr\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.026490 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.026396    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.127650 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.127612    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.127805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.127676    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.127805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.127716    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.127805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.127752    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.127805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.127766    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.127805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.127781    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.127805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.127796    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.128039 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.127820    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mnr\" (UniqueName: \"kubernetes.io/projected/f1ac3110-5846-48f8-a0d4-edf87b4defb1-kube-api-access-x9mnr\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.128039 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.127865    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.128238 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.128215    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.128708 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.128614    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.128708 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.128677    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.128923 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.128897    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.129183 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.129160    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.130527 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.130502    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.130962 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.130944    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.148962 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.148924    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mnr\" (UniqueName: \"kubernetes.io/projected/f1ac3110-5846-48f8-a0d4-edf87b4defb1-kube-api-access-x9mnr\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.149362 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.149341    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ac3110-5846-48f8-a0d4-edf87b4defb1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-tkhp2\" (UID: \"f1ac3110-5846-48f8-a0d4-edf87b4defb1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.278668 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.278634    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:18.617481 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:18.617457    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"]
Oct 30 02:46:18.620569 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:46:18.620537    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ac3110_5846_48f8_a0d4_edf87b4defb1.slice/crio-ffcf7fd96fe444abfd560727e10e455671ece9bd75a789a109db143cb5fe8fec WatchSource:0}: Error finding container ffcf7fd96fe444abfd560727e10e455671ece9bd75a789a109db143cb5fe8fec: Status 404 returned error can't find the container with id ffcf7fd96fe444abfd560727e10e455671ece9bd75a789a109db143cb5fe8fec
Oct 30 02:46:19.534860 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:19.534820    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2" event={"ID":"f1ac3110-5846-48f8-a0d4-edf87b4defb1","Type":"ContainerStarted","Data":"ffcf7fd96fe444abfd560727e10e455671ece9bd75a789a109db143cb5fe8fec"}
Oct 30 02:46:19.537084 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:19.537046    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-95rk5" event={"ID":"429df927-d84d-4d8c-9111-263b24400651","Type":"ContainerStarted","Data":"985202d52f4447455a218ef52ca8b7d36332e26aad8ab74f261a8329d78a4177"}
Oct 30 02:46:19.537212 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:19.537176    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:19.581400 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:19.581345    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-95rk5" podStartSLOduration=2.308465494 podStartE2EDuration="4.58132877s" totalImagesPullingTime="2.272863276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:46:19.579165851 +0000 UTC m=+317.503374041" watchObservedRunningTime="2026-10-30 02:46:19.58132877 +0000 UTC m=+317.505536958"
Oct 30 02:46:21.482806 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:21.482767    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:46:21.483202 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:21.482851    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:46:21.483202 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:21.482898    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:46:21.548118 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:21.548084    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2" event={"ID":"f1ac3110-5846-48f8-a0d4-edf87b4defb1","Type":"ContainerStarted","Data":"a8fbb76308396d907d4f9823c692dc0d4401757478073409ef6cecfc75d9ac8c"}
Oct 30 02:46:21.595130 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:21.595073    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2" podStartSLOduration=1.735963533 podStartE2EDuration="4.595057263s" totalImagesPullingTime="2.85909373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:46:21.594643 +0000 UTC m=+319.518851191" watchObservedRunningTime="2026-10-30 02:46:21.595057263 +0000 UTC m=+319.519265450"
Oct 30 02:46:22.279369 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:22.279321    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:22.284288 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:22.284264    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:22.552497 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:22.552402    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:22.553387 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:22.553370    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-tkhp2"
Oct 30 02:46:24.188906 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:46:24.188866    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:46:28.531331 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:28.531292    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-5vbrn"
Oct 30 02:46:30.545549 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:46:30.545519    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-95rk5"
Oct 30 02:46:54.187883 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:46:54.187838    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:47:13.789196 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.789159    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rbqcv"]
Oct 30 02:47:13.800029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.800001    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv"
Oct 30 02:47:13.813769 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.813743    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rbqcv"]
Oct 30 02:47:13.825522 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.825497    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 02:47:13.825734 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.825707    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lgn66\""
Oct 30 02:47:13.826887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.826866    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"]
Oct 30 02:47:13.830231 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.830211    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:47:13.848298 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.848271    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"]
Oct 30 02:47:13.875778 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.875753    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hhdnz"]
Oct 30 02:47:13.879011 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.878993    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz"
Oct 30 02:47:13.887680 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.887663    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 02:47:13.888654 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.888640    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kfrqb\""
Oct 30 02:47:13.899340 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.899319    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hhdnz"]
Oct 30 02:47:13.900963 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.900942    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhb2d\" (UniqueName: \"kubernetes.io/projected/6f0f3468-47c5-489c-9a4c-bf3279e50bdc-kube-api-access-dhb2d\") pod \"model-serving-api-75bd89fdfc-hhdnz\" (UID: \"6f0f3468-47c5-489c-9a4c-bf3279e50bdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz"
Oct 30 02:47:13.901050 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.900974    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvmc\" (UniqueName: \"kubernetes.io/projected/d1772e43-22c4-4f22-b106-704810d9d9cb-kube-api-access-nfvmc\") pod \"kserve-controller-manager-776cc66d47-rbqcv\" (UID: \"d1772e43-22c4-4f22-b106-704810d9d9cb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv"
Oct 30 02:47:13.901050 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.901010    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/369273db-efc8-4d7c-80ed-828877e8a6e5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-z54fn\" (UID: \"369273db-efc8-4d7c-80ed-828877e8a6e5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:47:13.901177 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.901152    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1772e43-22c4-4f22-b106-704810d9d9cb-cert\") pod \"kserve-controller-manager-776cc66d47-rbqcv\" (UID: \"d1772e43-22c4-4f22-b106-704810d9d9cb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv"
Oct 30 02:47:13.901227 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.901191    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0f3468-47c5-489c-9a4c-bf3279e50bdc-tls-certs\") pod \"model-serving-api-75bd89fdfc-hhdnz\" (UID: \"6f0f3468-47c5-489c-9a4c-bf3279e50bdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz"
Oct 30 02:47:13.901302 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.901285    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pj9d\" (UniqueName: \"kubernetes.io/projected/369273db-efc8-4d7c-80ed-828877e8a6e5-kube-api-access-6pj9d\") pod \"llmisvc-controller-manager-869d5dcc5d-z54fn\" (UID: \"369273db-efc8-4d7c-80ed-828877e8a6e5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:47:13.916602 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.916575    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-7qqcm"]
Oct 30 02:47:13.917259 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.917242    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-h9xc2\""
Oct 30 02:47:13.917347 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.917311    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 02:47:13.919988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.919970    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:13.939596 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:13.939575    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-7qqcm"]
Oct 30 02:47:14.001844 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.001813    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0f3468-47c5-489c-9a4c-bf3279e50bdc-tls-certs\") pod \"model-serving-api-75bd89fdfc-hhdnz\" (UID: \"6f0f3468-47c5-489c-9a4c-bf3279e50bdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz"
Oct 30 02:47:14.002031 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.001867    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4znz\" (UniqueName: \"kubernetes.io/projected/82bfdb57-cc86-4f29-aa87-43a3c3682105-kube-api-access-p4znz\") pod \"odh-model-controller-f87fcf56c-7qqcm\" (UID: \"82bfdb57-cc86-4f29-aa87-43a3c3682105\") " pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:14.002031 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.001909    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pj9d\" (UniqueName: \"kubernetes.io/projected/369273db-efc8-4d7c-80ed-828877e8a6e5-kube-api-access-6pj9d\") pod \"llmisvc-controller-manager-869d5dcc5d-z54fn\" (UID: \"369273db-efc8-4d7c-80ed-828877e8a6e5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:47:14.002031 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.001943    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhb2d\" (UniqueName: \"kubernetes.io/projected/6f0f3468-47c5-489c-9a4c-bf3279e50bdc-kube-api-access-dhb2d\") pod \"model-serving-api-75bd89fdfc-hhdnz\" (UID: \"6f0f3468-47c5-489c-9a4c-bf3279e50bdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz"
Oct 30 02:47:14.002031 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.002011    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvmc\" (UniqueName: \"kubernetes.io/projected/d1772e43-22c4-4f22-b106-704810d9d9cb-kube-api-access-nfvmc\") pod \"kserve-controller-manager-776cc66d47-rbqcv\" (UID: \"d1772e43-22c4-4f22-b106-704810d9d9cb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv"
Oct 30 02:47:14.002260 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.002038    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82bfdb57-cc86-4f29-aa87-43a3c3682105-cert\") pod \"odh-model-controller-f87fcf56c-7qqcm\" (UID: \"82bfdb57-cc86-4f29-aa87-43a3c3682105\") " pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:14.002260 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.002081    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/369273db-efc8-4d7c-80ed-828877e8a6e5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-z54fn\" (UID: \"369273db-efc8-4d7c-80ed-828877e8a6e5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:47:14.002260 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.002108    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82bfdb57-cc86-4f29-aa87-43a3c3682105-metrics-certs\") pod \"odh-model-controller-f87fcf56c-7qqcm\" (UID: \"82bfdb57-cc86-4f29-aa87-43a3c3682105\") " pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:14.002260 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.002194    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1772e43-22c4-4f22-b106-704810d9d9cb-cert\") pod \"kserve-controller-manager-776cc66d47-rbqcv\" (UID: \"d1772e43-22c4-4f22-b106-704810d9d9cb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv"
Oct 30 02:47:14.004316 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.004295    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0f3468-47c5-489c-9a4c-bf3279e50bdc-tls-certs\") pod \"model-serving-api-75bd89fdfc-hhdnz\" (UID: \"6f0f3468-47c5-489c-9a4c-bf3279e50bdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz"
Oct 30 02:47:14.004479 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.004460    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1772e43-22c4-4f22-b106-704810d9d9cb-cert\") pod \"kserve-controller-manager-776cc66d47-rbqcv\" (UID: \"d1772e43-22c4-4f22-b106-704810d9d9cb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv"
Oct 30 02:47:14.004563 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.004545    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/369273db-efc8-4d7c-80ed-828877e8a6e5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-z54fn\" (UID: \"369273db-efc8-4d7c-80ed-828877e8a6e5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:47:14.017802 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.017784    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fnzpw\""
Oct 30 02:47:14.017802 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.017786    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 02:47:14.017935 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.017786    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 02:47:14.022097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.022072    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvmc\" (UniqueName: \"kubernetes.io/projected/d1772e43-22c4-4f22-b106-704810d9d9cb-kube-api-access-nfvmc\") pod \"kserve-controller-manager-776cc66d47-rbqcv\" (UID: \"d1772e43-22c4-4f22-b106-704810d9d9cb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv"
Oct 30 02:47:14.022097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.022088    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pj9d\" (UniqueName: \"kubernetes.io/projected/369273db-efc8-4d7c-80ed-828877e8a6e5-kube-api-access-6pj9d\") pod \"llmisvc-controller-manager-869d5dcc5d-z54fn\" (UID: \"369273db-efc8-4d7c-80ed-828877e8a6e5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:47:14.022253 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.022103    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhb2d\" (UniqueName: \"kubernetes.io/projected/6f0f3468-47c5-489c-9a4c-bf3279e50bdc-kube-api-access-dhb2d\") pod \"model-serving-api-75bd89fdfc-hhdnz\" (UID: \"6f0f3468-47c5-489c-9a4c-bf3279e50bdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz"
Oct 30 02:47:14.103375 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.103301    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82bfdb57-cc86-4f29-aa87-43a3c3682105-metrics-certs\") pod \"odh-model-controller-f87fcf56c-7qqcm\" (UID: \"82bfdb57-cc86-4f29-aa87-43a3c3682105\") " pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:14.103523 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.103374    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4znz\" (UniqueName: \"kubernetes.io/projected/82bfdb57-cc86-4f29-aa87-43a3c3682105-kube-api-access-p4znz\") pod \"odh-model-controller-f87fcf56c-7qqcm\" (UID: \"82bfdb57-cc86-4f29-aa87-43a3c3682105\") " pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:14.103523 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.103450    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82bfdb57-cc86-4f29-aa87-43a3c3682105-cert\") pod \"odh-model-controller-f87fcf56c-7qqcm\" (UID: \"82bfdb57-cc86-4f29-aa87-43a3c3682105\") " pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:14.105732 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.105708    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82bfdb57-cc86-4f29-aa87-43a3c3682105-cert\") pod \"odh-model-controller-f87fcf56c-7qqcm\" (UID: \"82bfdb57-cc86-4f29-aa87-43a3c3682105\") " pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:14.105732 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.105726    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82bfdb57-cc86-4f29-aa87-43a3c3682105-metrics-certs\") pod \"odh-model-controller-f87fcf56c-7qqcm\" (UID: \"82bfdb57-cc86-4f29-aa87-43a3c3682105\") " pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:14.112617 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.112597    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv"
Oct 30 02:47:14.124242 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.124216    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4znz\" (UniqueName: \"kubernetes.io/projected/82bfdb57-cc86-4f29-aa87-43a3c3682105-kube-api-access-p4znz\") pod \"odh-model-controller-f87fcf56c-7qqcm\" (UID: \"82bfdb57-cc86-4f29-aa87-43a3c3682105\") " pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:14.144205 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.144172    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:47:14.192360 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.192253    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz"
Oct 30 02:47:14.233679 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.232708    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:14.421755 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.421730    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rbqcv"]
Oct 30 02:47:14.423379 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:47:14.423350    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1772e43_22c4_4f22_b106_704810d9d9cb.slice/crio-629f5459ce2d11dcdbc91a6c9578c4865f60c32ad16b0a1e11b9aab43a6a4248 WatchSource:0}: Error finding container 629f5459ce2d11dcdbc91a6c9578c4865f60c32ad16b0a1e11b9aab43a6a4248: Status 404 returned error can't find the container with id 629f5459ce2d11dcdbc91a6c9578c4865f60c32ad16b0a1e11b9aab43a6a4248
Oct 30 02:47:14.459826 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.459804    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"]
Oct 30 02:47:14.462205 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:47:14.462180    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod369273db_efc8_4d7c_80ed_828877e8a6e5.slice/crio-34bad282f6d4a5ac2e4bb96a902a484decc0b40eb18d023fee60c1d3fb3efffd WatchSource:0}: Error finding container 34bad282f6d4a5ac2e4bb96a902a484decc0b40eb18d023fee60c1d3fb3efffd: Status 404 returned error can't find the container with id 34bad282f6d4a5ac2e4bb96a902a484decc0b40eb18d023fee60c1d3fb3efffd
Oct 30 02:47:14.503398 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.503341    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hhdnz"]
Oct 30 02:47:14.505802 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:47:14.505780    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0f3468_47c5_489c_9a4c_bf3279e50bdc.slice/crio-728cb3a20f4170b04521ce7da35789865ecbcd5da0dd9e94786c5c63788be3c1 WatchSource:0}: Error finding container 728cb3a20f4170b04521ce7da35789865ecbcd5da0dd9e94786c5c63788be3c1: Status 404 returned error can't find the container with id 728cb3a20f4170b04521ce7da35789865ecbcd5da0dd9e94786c5c63788be3c1
Oct 30 02:47:14.536062 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.535985    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-7qqcm"]
Oct 30 02:47:14.537884 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:47:14.537864    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bfdb57_cc86_4f29_aa87_43a3c3682105.slice/crio-edee4d43ee6ec37b90a0acf3ee9f0c5f06cb244ff6386a01f560baf9f5a288c0 WatchSource:0}: Error finding container edee4d43ee6ec37b90a0acf3ee9f0c5f06cb244ff6386a01f560baf9f5a288c0: Status 404 returned error can't find the container with id edee4d43ee6ec37b90a0acf3ee9f0c5f06cb244ff6386a01f560baf9f5a288c0
Oct 30 02:47:14.769058 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.769024    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz" event={"ID":"6f0f3468-47c5-489c-9a4c-bf3279e50bdc","Type":"ContainerStarted","Data":"728cb3a20f4170b04521ce7da35789865ecbcd5da0dd9e94786c5c63788be3c1"}
Oct 30 02:47:14.770143 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.770120    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm" event={"ID":"82bfdb57-cc86-4f29-aa87-43a3c3682105","Type":"ContainerStarted","Data":"edee4d43ee6ec37b90a0acf3ee9f0c5f06cb244ff6386a01f560baf9f5a288c0"}
Oct 30 02:47:14.771245 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.771224    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn" event={"ID":"369273db-efc8-4d7c-80ed-828877e8a6e5","Type":"ContainerStarted","Data":"34bad282f6d4a5ac2e4bb96a902a484decc0b40eb18d023fee60c1d3fb3efffd"}
Oct 30 02:47:14.772224 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:14.772197    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv" event={"ID":"d1772e43-22c4-4f22-b106-704810d9d9cb","Type":"ContainerStarted","Data":"629f5459ce2d11dcdbc91a6c9578c4865f60c32ad16b0a1e11b9aab43a6a4248"}
Oct 30 02:47:17.797018 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:17.796961    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz" event={"ID":"6f0f3468-47c5-489c-9a4c-bf3279e50bdc","Type":"ContainerStarted","Data":"bdb84398038465af3b4e2cfffc1d386e8f60afb94ae69237d4effe44367f3c0c"}
Oct 30 02:47:17.797452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:17.797106    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz"
Oct 30 02:47:18.803028 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:18.802993    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn" event={"ID":"369273db-efc8-4d7c-80ed-828877e8a6e5","Type":"ContainerStarted","Data":"566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e"}
Oct 30 02:47:18.803374 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:18.803321    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:47:18.845685 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:18.845633    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn" podStartSLOduration=1.6281627950000002 podStartE2EDuration="5.845619061s" totalImagesPullingTime="4.217456266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:47:18.845123881 +0000 UTC m=+376.769332069" watchObservedRunningTime="2026-10-30 02:47:18.845619061 +0000 UTC m=+376.769827299"
Oct 30 02:47:18.846394 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:18.846364    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz" podStartSLOduration=2.710340489 podStartE2EDuration="5.846356032s" totalImagesPullingTime="3.136015543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:47:17.840616311 +0000 UTC m=+375.764824502" watchObservedRunningTime="2026-10-30 02:47:18.846356032 +0000 UTC m=+376.770564222"
Oct 30 02:47:19.809470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:19.809415    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm" event={"ID":"82bfdb57-cc86-4f29-aa87-43a3c3682105","Type":"ContainerStarted","Data":"fc5452a8d7267deff48a3263adbd3cba304bc4450d6de66c51b40e8cf1ca49bc"}
Oct 30 02:47:19.810029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:19.809570    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:19.811059 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:19.811035    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv" event={"ID":"d1772e43-22c4-4f22-b106-704810d9d9cb","Type":"ContainerStarted","Data":"e62f7bc1cd2c50dc823c4b36c244b8904d9fba1765817cb953c5b21078887ab6"}
Oct 30 02:47:19.811179 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:19.811165    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv"
Oct 30 02:47:19.850802 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:19.850754    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm" podStartSLOduration=2.640262329 podStartE2EDuration="6.850738022s" totalImagesPullingTime="4.210475693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:47:19.849719756 +0000 UTC m=+377.773927947" watchObservedRunningTime="2026-10-30 02:47:19.850738022 +0000 UTC m=+377.774946231"
Oct 30 02:47:19.891266 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:19.891214    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv" podStartSLOduration=2.492498111 podStartE2EDuration="6.891200431s" totalImagesPullingTime="4.39870232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:47:19.889953017 +0000 UTC m=+377.814161217" watchObservedRunningTime="2026-10-30 02:47:19.891200431 +0000 UTC m=+377.815408619"
Oct 30 02:47:24.188513 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:47:24.188476    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:47:28.807766 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:28.807737    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-hhdnz"
Oct 30 02:47:30.817765 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:30.817735    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-7qqcm"
Oct 30 02:47:49.813532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:49.813457    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:47:50.820576 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:47:50.820541    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-rbqcv"
Oct 30 02:47:54.188968 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:47:54.188932    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:48:24.173289 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.173258    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"]
Oct 30 02:48:24.175635 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.175612    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:24.189526 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:24.189501    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:48:24.193828 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.193808    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 02:48:24.203659 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.203641    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 02:48:24.207859 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.207828    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2lmpt\""
Oct 30 02:48:24.218928 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.218901    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"]
Oct 30 02:48:24.351489 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.351456    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvvd\" (UniqueName: \"kubernetes.io/projected/ba815604-2713-4214-b8e3-cf7c4411accf-kube-api-access-pxvvd\") pod \"limitador-operator-controller-manager-856dd584c5-9bkgp\" (UID: \"ba815604-2713-4214-b8e3-cf7c4411accf\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:24.452444 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.452354    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvvd\" (UniqueName: \"kubernetes.io/projected/ba815604-2713-4214-b8e3-cf7c4411accf-kube-api-access-pxvvd\") pod \"limitador-operator-controller-manager-856dd584c5-9bkgp\" (UID: \"ba815604-2713-4214-b8e3-cf7c4411accf\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:24.476351 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.476327    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvvd\" (UniqueName: \"kubernetes.io/projected/ba815604-2713-4214-b8e3-cf7c4411accf-kube-api-access-pxvvd\") pod \"limitador-operator-controller-manager-856dd584c5-9bkgp\" (UID: \"ba815604-2713-4214-b8e3-cf7c4411accf\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:24.488114 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.488096    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:24.781386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:24.781365    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"]
Oct 30 02:48:24.783762 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:48:24.783732    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba815604_2713_4214_b8e3_cf7c4411accf.slice/crio-b232ae9697309f53bb7e48394fead200909b722719ac105a6df00b7c9368dd4c WatchSource:0}: Error finding container b232ae9697309f53bb7e48394fead200909b722719ac105a6df00b7c9368dd4c: Status 404 returned error can't find the container with id b232ae9697309f53bb7e48394fead200909b722719ac105a6df00b7c9368dd4c
Oct 30 02:48:25.117756 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:25.117676    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp" event={"ID":"ba815604-2713-4214-b8e3-cf7c4411accf","Type":"ContainerStarted","Data":"b232ae9697309f53bb7e48394fead200909b722719ac105a6df00b7c9368dd4c"}
Oct 30 02:48:27.128123 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:27.128093    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp" event={"ID":"ba815604-2713-4214-b8e3-cf7c4411accf","Type":"ContainerStarted","Data":"d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89"}
Oct 30 02:48:27.128549 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:27.128187    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:27.171545 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:27.171492    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp" podStartSLOduration=0.914471967 podStartE2EDuration="3.171478306s" totalImagesPullingTime="2.257006339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:27.170091839 +0000 UTC m=+445.094300028" watchObservedRunningTime="2026-10-30 02:48:27.171478306 +0000 UTC m=+445.095686494"
Oct 30 02:48:38.134194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:38.134165    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:39.827257 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:39.827219    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"]
Oct 30 02:48:39.831082 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:39.831062    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:39.846291 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:39.846269    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"]
Oct 30 02:48:39.858355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:39.858336    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-g7ggq\""
Oct 30 02:48:39.885821 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:39.885795    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-m7mlc\" (UID: \"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:39.885935 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:39.885887    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klkjk\" (UniqueName: \"kubernetes.io/projected/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-kube-api-access-klkjk\") pod \"kuadrant-operator-controller-manager-956956bc4-m7mlc\" (UID: \"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:39.986907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:39.986876    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-m7mlc\" (UID: \"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:39.987050 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:39.986910    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klkjk\" (UniqueName: \"kubernetes.io/projected/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-kube-api-access-klkjk\") pod \"kuadrant-operator-controller-manager-956956bc4-m7mlc\" (UID: \"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:39.987249 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:39.987229    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-m7mlc\" (UID: \"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:40.008630 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:40.008605    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klkjk\" (UniqueName: \"kubernetes.io/projected/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-kube-api-access-klkjk\") pod \"kuadrant-operator-controller-manager-956956bc4-m7mlc\" (UID: \"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:40.143987 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:40.143905    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:40.468502 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:40.468479    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"]
Oct 30 02:48:40.470528 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:48:40.470490    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fc9fc9_ab7a_45ff_91c5_efa6d3cdecaa.slice/crio-8d287a1578e743126001042121837e04b436755276d0fd5160284d36086f7ebd WatchSource:0}: Error finding container 8d287a1578e743126001042121837e04b436755276d0fd5160284d36086f7ebd: Status 404 returned error can't find the container with id 8d287a1578e743126001042121837e04b436755276d0fd5160284d36086f7ebd
Oct 30 02:48:41.188748 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.188711    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc" event={"ID":"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa","Type":"ContainerStarted","Data":"8d287a1578e743126001042121837e04b436755276d0fd5160284d36086f7ebd"}
Oct 30 02:48:41.512810 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.512779    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"]
Oct 30 02:48:41.512992 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.512844    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"]
Oct 30 02:48:41.526811 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.526785    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"]
Oct 30 02:48:41.527061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.527027    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp" podUID="ba815604-2713-4214-b8e3-cf7c4411accf" containerName="manager" containerID="cri-o://d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89" gracePeriod=2
Oct 30 02:48:41.542351 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.542319    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"]
Oct 30 02:48:41.542503 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:41.542463    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-9bkgp\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="ba815604-2713-4214-b8e3-cf7c4411accf" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:41.566461 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.566412    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td"]
Oct 30 02:48:41.567029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.567005    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ba815604-2713-4214-b8e3-cf7c4411accf" containerName="manager"
Oct 30 02:48:41.567029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.567031    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba815604-2713-4214-b8e3-cf7c4411accf" containerName="manager"
Oct 30 02:48:41.567225 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.567168    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba815604-2713-4214-b8e3-cf7c4411accf" containerName="manager"
Oct 30 02:48:41.571838 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.571803    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td"
Oct 30 02:48:41.587181 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:41.587143    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-9bkgp\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="ba815604-2713-4214-b8e3-cf7c4411accf" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:41.591319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.591288    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td"]
Oct 30 02:48:41.600487 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.600454    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqqfh\" (UniqueName: \"kubernetes.io/projected/b28fd796-8c22-49dd-92d4-714303b3ae5e-kube-api-access-xqqfh\") pod \"limitador-operator-controller-manager-856dd584c5-xv2td\" (UID: \"b28fd796-8c22-49dd-92d4-714303b3ae5e\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td"
Oct 30 02:48:41.702143 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.702107    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqqfh\" (UniqueName: \"kubernetes.io/projected/b28fd796-8c22-49dd-92d4-714303b3ae5e-kube-api-access-xqqfh\") pod \"limitador-operator-controller-manager-856dd584c5-xv2td\" (UID: \"b28fd796-8c22-49dd-92d4-714303b3ae5e\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td"
Oct 30 02:48:41.727763 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.727733    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqqfh\" (UniqueName: \"kubernetes.io/projected/b28fd796-8c22-49dd-92d4-714303b3ae5e-kube-api-access-xqqfh\") pod \"limitador-operator-controller-manager-856dd584c5-xv2td\" (UID: \"b28fd796-8c22-49dd-92d4-714303b3ae5e\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td"
Oct 30 02:48:41.801356 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.801331    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:41.816555 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:41.816524    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-9bkgp\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="ba815604-2713-4214-b8e3-cf7c4411accf" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:41.904771 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.904734    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba815604-2713-4214-b8e3-cf7c4411accf-kube-api-access-pxvvd\" (UniqueName: \"kubernetes.io/projected/ba815604-2713-4214-b8e3-cf7c4411accf-kube-api-access-pxvvd\") pod \"ba815604-2713-4214-b8e3-cf7c4411accf\" (UID: \"ba815604-2713-4214-b8e3-cf7c4411accf\") "
Oct 30 02:48:41.907284 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.907258    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba815604-2713-4214-b8e3-cf7c4411accf-kube-api-access-pxvvd" pod "ba815604-2713-4214-b8e3-cf7c4411accf" (UID: "ba815604-2713-4214-b8e3-cf7c4411accf"). InnerVolumeSpecName "kube-api-access-pxvvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:48:41.908334 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:41.908312    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td"
Oct 30 02:48:42.005486 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:42.005450    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxvvd\" (UniqueName: \"kubernetes.io/projected/ba815604-2713-4214-b8e3-cf7c4411accf-kube-api-access-pxvvd\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:48:42.196960 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:42.196876    4278 generic.go:356] "Generic (PLEG): container finished" podID="ba815604-2713-4214-b8e3-cf7c4411accf" containerID="d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89" exitCode=0
Oct 30 02:48:42.197414 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:42.196971    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:42.200537 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:42.200507    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba815604-2713-4214-b8e3-cf7c4411accf" path="/var/lib/kubelet/pods/ba815604-2713-4214-b8e3-cf7c4411accf/volumes"
Oct 30 02:48:42.200986 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:42.200969    4278 scope.go:122] "RemoveContainer" containerID="d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89"
Oct 30 02:48:42.209605 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:42.209019    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-9bkgp\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="ba815604-2713-4214-b8e3-cf7c4411accf" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-9bkgp"
Oct 30 02:48:42.211977 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:42.211367    4278 scope.go:122] "RemoveContainer" containerID="d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89"
Oct 30 02:48:42.213613 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:42.213594    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td"]
Oct 30 02:48:42.213916 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:42.213893    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89\": container with ID starting with d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89 not found: ID does not exist" containerID="d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89"
Oct 30 02:48:42.213993 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:42.213927    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89"} err="failed to get container status \"d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89\": rpc error: code = NotFound desc = could not find container \"d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89\": container with ID starting with d3900ba71ea6f22780edc524cd9a301af1e85fade5109d00ba11e87d4b07ce89 not found: ID does not exist"
Oct 30 02:48:42.215878 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:48:42.215858    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28fd796_8c22_49dd_92d4_714303b3ae5e.slice/crio-386a3aa554e0d93c7af5dd8080f69f3fd99c2736b461effd2032bc71842193e3 WatchSource:0}: Error finding container 386a3aa554e0d93c7af5dd8080f69f3fd99c2736b461effd2032bc71842193e3: Status 404 returned error can't find the container with id 386a3aa554e0d93c7af5dd8080f69f3fd99c2736b461effd2032bc71842193e3
Oct 30 02:48:43.204184 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:43.204142    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td" event={"ID":"b28fd796-8c22-49dd-92d4-714303b3ae5e","Type":"ContainerStarted","Data":"652c47b5b7406c3174714d7dd242461b9c1e21ba1df1eb0ab425bc8db92350d0"}
Oct 30 02:48:43.204614 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:43.204240    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td" event={"ID":"b28fd796-8c22-49dd-92d4-714303b3ae5e","Type":"ContainerStarted","Data":"386a3aa554e0d93c7af5dd8080f69f3fd99c2736b461effd2032bc71842193e3"}
Oct 30 02:48:43.204614 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:43.204270    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td"
Oct 30 02:48:43.247179 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:43.247130    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td" podStartSLOduration=2.247117279 podStartE2EDuration="2.247117279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:43.245604737 +0000 UTC m=+461.169812925" watchObservedRunningTime="2026-10-30 02:48:43.247117279 +0000 UTC m=+461.171325467"
Oct 30 02:48:44.210966 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:44.210868    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc" podUID="71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" containerName="manager" containerID="cri-o://e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38" gracePeriod=2
Oct 30 02:48:44.461759 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:44.461700    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:44.476312 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:44.476288    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-m7mlc\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:44.529510 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:44.529474    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-extensions-socket-volume\") pod \"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa\" (UID: \"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa\") "
Oct 30 02:48:44.529510 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:44.529519    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-kube-api-access-klkjk\" (UniqueName: \"kubernetes.io/projected/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-kube-api-access-klkjk\") pod \"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa\" (UID: \"71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa\") "
Oct 30 02:48:44.529712 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:44.529692    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-extensions-socket-volume" pod "71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" (UID: "71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:44.531535 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:44.531514    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-kube-api-access-klkjk" pod "71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" (UID: "71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa"). InnerVolumeSpecName "kube-api-access-klkjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:48:44.630679 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:44.630642    4278 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-extensions-socket-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:48:44.630679 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:44.630677    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klkjk\" (UniqueName: \"kubernetes.io/projected/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa-kube-api-access-klkjk\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:48:45.216286 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:45.216255    4278 generic.go:356] "Generic (PLEG): container finished" podID="71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" containerID="e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38" exitCode=2
Oct 30 02:48:45.216733 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:45.216296    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:45.216733 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:45.216354    4278 scope.go:122] "RemoveContainer" containerID="e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38"
Oct 30 02:48:45.224967 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:45.224945    4278 scope.go:122] "RemoveContainer" containerID="e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38"
Oct 30 02:48:45.225213 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:45.225195    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38\": container with ID starting with e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38 not found: ID does not exist" containerID="e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38"
Oct 30 02:48:45.225281 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:45.225227    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38"} err="failed to get container status \"e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38\": rpc error: code = NotFound desc = could not find container \"e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38\": container with ID starting with e70b9f10a163e364721edc665ab0afc71b63529d208b9c6d86e4ca7fe05adb38 not found: ID does not exist"
Oct 30 02:48:45.231418 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:45.231395    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-m7mlc\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:45.246032 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:45.246007    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-m7mlc\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-m7mlc"
Oct 30 02:48:46.197977 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:46.197945    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" path="/var/lib/kubelet/pods/71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa/volumes"
Oct 30 02:48:54.188158 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:48:54.188121    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:48:54.212753 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:48:54.212729    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-xv2td"
Oct 30 02:49:11.205504 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.205469    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"]
Oct 30 02:49:11.206184 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.206160    4278 cpu_manager.go:414] "containerMap: removing container" podUID="71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" containerName="manager"
Oct 30 02:49:11.206247 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.206188    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" containerName="manager"
Oct 30 02:49:11.206292 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.206281    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fc9fc9-ab7a-45ff-91c5-efa6d3cdecaa" containerName="manager"
Oct 30 02:49:11.214635 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.214608    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.236861 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.236833    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"]
Oct 30 02:49:11.247668 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.247642    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wg9w9\""
Oct 30 02:49:11.369484 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.369444    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ea4316-7b4f-4453-a930-3aea23f78fb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.369703 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.369539    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.369703 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.369619    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t84nq\" (UniqueName: \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-kube-api-access-t84nq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.369703 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.369648    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.369703 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.369678    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.369892 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.369761    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.369892 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.369786    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.369892 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.369807    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.369892 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.369827    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471111 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471013    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471111 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471062    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471329 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471213    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471329 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471284    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471473 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471371    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ea4316-7b4f-4453-a930-3aea23f78fb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471473 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471409    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471474    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t84nq\" (UniqueName: \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-kube-api-access-t84nq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471507    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471524    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471750 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471728    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.471814 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.471784    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.472048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.472024    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.472160 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.472142    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.472160 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.472152    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ea4316-7b4f-4453-a930-3aea23f78fb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.473927 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.473896    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.474070 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.474051    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.495239 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.495207    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.495384 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.495353    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t84nq\" (UniqueName: \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-kube-api-access-t84nq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w7qxw\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.529340 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.529312    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:11.838639 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.838610    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"]
Oct 30 02:49:11.840960 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:49:11.840932    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ea4316_7b4f_4453_a930_3aea23f78fb1.slice/crio-dc0a3dcba82ace132701d394a065f2d5e4efb20766b306ec3bae852a50a8a0d7 WatchSource:0}: Error finding container dc0a3dcba82ace132701d394a065f2d5e4efb20766b306ec3bae852a50a8a0d7: Status 404 returned error can't find the container with id dc0a3dcba82ace132701d394a065f2d5e4efb20766b306ec3bae852a50a8a0d7
Oct 30 02:49:11.842965 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.842938    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:49:11.843031 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.842999    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:49:11.843031 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:11.843028    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:49:12.334383 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:12.334347    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" event={"ID":"76ea4316-7b4f-4453-a930-3aea23f78fb1","Type":"ContainerStarted","Data":"26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790"}
Oct 30 02:49:12.334939 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:12.334409    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" event={"ID":"76ea4316-7b4f-4453-a930-3aea23f78fb1","Type":"ContainerStarted","Data":"dc0a3dcba82ace132701d394a065f2d5e4efb20766b306ec3bae852a50a8a0d7"}
Oct 30 02:49:12.405087 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:12.405033    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" podStartSLOduration=1.4050186390000001 podStartE2EDuration="1.405018639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:12.403456432 +0000 UTC m=+490.327664618" watchObservedRunningTime="2026-10-30 02:49:12.405018639 +0000 UTC m=+490.329226826"
Oct 30 02:49:12.529960 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:12.529928    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:12.534575 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:12.534554    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:13.338495 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:13.338462    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:13.339856 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:13.339834    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:49:24.187482 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:49:24.187449    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:49:26.321491 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.321457    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:49:26.323888 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.323869    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s"
Oct 30 02:49:26.348579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.348553    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:49:26.359742 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.359716    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 02:49:26.359880 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.359814    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-88pp5\""
Oct 30 02:49:26.407343 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.407314    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:49:26.498216 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.498181    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9617029d-abc2-416a-9628-0e25453cafe7-config-file\") pod \"limitador-limitador-7c9b87d574-vx95s\" (UID: \"9617029d-abc2-416a-9628-0e25453cafe7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s"
Oct 30 02:49:26.498216 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.498217    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpfmj\" (UniqueName: \"kubernetes.io/projected/9617029d-abc2-416a-9628-0e25453cafe7-kube-api-access-fpfmj\") pod \"limitador-limitador-7c9b87d574-vx95s\" (UID: \"9617029d-abc2-416a-9628-0e25453cafe7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s"
Oct 30 02:49:26.599389 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.599291    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9617029d-abc2-416a-9628-0e25453cafe7-config-file\") pod \"limitador-limitador-7c9b87d574-vx95s\" (UID: \"9617029d-abc2-416a-9628-0e25453cafe7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s"
Oct 30 02:49:26.599389 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.599336    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpfmj\" (UniqueName: \"kubernetes.io/projected/9617029d-abc2-416a-9628-0e25453cafe7-kube-api-access-fpfmj\") pod \"limitador-limitador-7c9b87d574-vx95s\" (UID: \"9617029d-abc2-416a-9628-0e25453cafe7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s"
Oct 30 02:49:26.599949 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.599929    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9617029d-abc2-416a-9628-0e25453cafe7-config-file\") pod \"limitador-limitador-7c9b87d574-vx95s\" (UID: \"9617029d-abc2-416a-9628-0e25453cafe7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s"
Oct 30 02:49:26.621126 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.621101    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpfmj\" (UniqueName: \"kubernetes.io/projected/9617029d-abc2-416a-9628-0e25453cafe7-kube-api-access-fpfmj\") pod \"limitador-limitador-7c9b87d574-vx95s\" (UID: \"9617029d-abc2-416a-9628-0e25453cafe7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s"
Oct 30 02:49:26.636905 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.636886    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s"
Oct 30 02:49:26.931085 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:26.931064    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:49:26.933076 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:49:26.933050    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9617029d_abc2_416a_9628_0e25453cafe7.slice/crio-e119aa0f13f166e49bc9aea024a6569064b23bc7758ba7e939a50f97a23463e8 WatchSource:0}: Error finding container e119aa0f13f166e49bc9aea024a6569064b23bc7758ba7e939a50f97a23463e8: Status 404 returned error can't find the container with id e119aa0f13f166e49bc9aea024a6569064b23bc7758ba7e939a50f97a23463e8
Oct 30 02:49:27.397964 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:27.397933    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s" event={"ID":"9617029d-abc2-416a-9628-0e25453cafe7","Type":"ContainerStarted","Data":"e119aa0f13f166e49bc9aea024a6569064b23bc7758ba7e939a50f97a23463e8"}
Oct 30 02:49:28.403871 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:28.403840    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s" event={"ID":"9617029d-abc2-416a-9628-0e25453cafe7","Type":"ContainerStarted","Data":"923b563812b49bc9af28dc8bbc3d5a8138291ddd98374949f182535f845e8477"}
Oct 30 02:49:28.404248 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:28.403979    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s"
Oct 30 02:49:28.448851 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:28.448805    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s" podStartSLOduration=1.644204258 podStartE2EDuration="2.448793176s" totalImagesPullingTime="804.588918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:28.447888071 +0000 UTC m=+506.372096258" watchObservedRunningTime="2026-10-30 02:49:28.448793176 +0000 UTC m=+506.373001363"
Oct 30 02:49:39.409635 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:49:39.409606    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-vx95s"
Oct 30 02:49:54.187728 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:49:54.187689    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:50:24.187968 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:50:24.187935    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:50:47.699022 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:47.698936    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:50:54.186709 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:50:54.186629    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:50:56.464854 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.464823    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-474bl"]
Oct 30 02:50:56.467057 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.467042    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl"
Oct 30 02:50:56.487476 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.487404    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-474bl"]
Oct 30 02:50:56.574198 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.574169    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6jwc\" (UniqueName: \"kubernetes.io/projected/f6d27d9d-a11d-4f43-9c83-02658f4b0079-kube-api-access-x6jwc\") pod \"llmisvc-controller-manager-6466c554d8-474bl\" (UID: \"f6d27d9d-a11d-4f43-9c83-02658f4b0079\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl"
Oct 30 02:50:56.574352 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.574301    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6d27d9d-a11d-4f43-9c83-02658f4b0079-cert\") pod \"llmisvc-controller-manager-6466c554d8-474bl\" (UID: \"f6d27d9d-a11d-4f43-9c83-02658f4b0079\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl"
Oct 30 02:50:56.675559 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.675498    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6d27d9d-a11d-4f43-9c83-02658f4b0079-cert\") pod \"llmisvc-controller-manager-6466c554d8-474bl\" (UID: \"f6d27d9d-a11d-4f43-9c83-02658f4b0079\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl"
Oct 30 02:50:56.675559 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.675534    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6jwc\" (UniqueName: \"kubernetes.io/projected/f6d27d9d-a11d-4f43-9c83-02658f4b0079-kube-api-access-x6jwc\") pod \"llmisvc-controller-manager-6466c554d8-474bl\" (UID: \"f6d27d9d-a11d-4f43-9c83-02658f4b0079\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl"
Oct 30 02:50:56.677770 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.677742    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6d27d9d-a11d-4f43-9c83-02658f4b0079-cert\") pod \"llmisvc-controller-manager-6466c554d8-474bl\" (UID: \"f6d27d9d-a11d-4f43-9c83-02658f4b0079\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl"
Oct 30 02:50:56.695624 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.695599    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6jwc\" (UniqueName: \"kubernetes.io/projected/f6d27d9d-a11d-4f43-9c83-02658f4b0079-kube-api-access-x6jwc\") pod \"llmisvc-controller-manager-6466c554d8-474bl\" (UID: \"f6d27d9d-a11d-4f43-9c83-02658f4b0079\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl"
Oct 30 02:50:56.778539 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.778513    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl"
Oct 30 02:50:56.806548 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.806522    4278 generic.go:356] "Generic (PLEG): container finished" podID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerID="26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790" exitCode=0
Oct 30 02:50:56.806645 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.806592    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" event={"ID":"76ea4316-7b4f-4453-a930-3aea23f78fb1","Type":"ContainerDied","Data":"26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790"}
Oct 30 02:50:56.806975 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.806959    4278 scope.go:122] "RemoveContainer" containerID="26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790"
Oct 30 02:50:56.808138 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.808114    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:50:56.808197 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.808174    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:50:56.808230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:56.808209    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:50:57.072109 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:57.072032    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-474bl"]
Oct 30 02:50:57.075015 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:50:57.074990    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d27d9d_a11d_4f43_9c83_02658f4b0079.slice/crio-ef3821b033d4c148422b241822a70a3ab9a2dcb219657d9186ea84f02e845851 WatchSource:0}: Error finding container ef3821b033d4c148422b241822a70a3ab9a2dcb219657d9186ea84f02e845851: Status 404 returned error can't find the container with id ef3821b033d4c148422b241822a70a3ab9a2dcb219657d9186ea84f02e845851
Oct 30 02:50:57.812957 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:57.812919    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl" event={"ID":"f6d27d9d-a11d-4f43-9c83-02658f4b0079","Type":"ContainerStarted","Data":"45848b98a91a68f7b79c3cce34f2354f38d0f1b3de1eda59687911e7bab38558"}
Oct 30 02:50:57.813454 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:57.813007    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl" event={"ID":"f6d27d9d-a11d-4f43-9c83-02658f4b0079","Type":"ContainerStarted","Data":"ef3821b033d4c148422b241822a70a3ab9a2dcb219657d9186ea84f02e845851"}
Oct 30 02:50:57.813454 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:57.813040    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl"
Oct 30 02:50:57.814965 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:57.814940    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" event={"ID":"76ea4316-7b4f-4453-a930-3aea23f78fb1","Type":"ContainerStarted","Data":"3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318"}
Oct 30 02:50:57.854566 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:57.854509    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl" podStartSLOduration=1.4933853240000001 podStartE2EDuration="1.854493151s" totalImagesPullingTime="361.107827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:50:57.853761943 +0000 UTC m=+595.777970131" watchObservedRunningTime="2026-10-30 02:50:57.854493151 +0000 UTC m=+595.778701356"
Oct 30 02:50:58.338822 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:58.338790    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:50:58.529826 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:58.529786    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:50:58.530818 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:58.530789    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 02:50:58.530940 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:58.530840    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 02:50:59.529865 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:59.529831    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 02:50:59.530277 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:50:59.529881    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 02:51:00.530378 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:00.530342    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 02:51:00.530808 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:00.530392    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 02:51:01.530585 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:01.530554    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 02:51:01.530936 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:01.530604    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 02:51:02.530133 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:02.530081    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 02:51:02.530351 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:02.530140    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 02:51:03.533948 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:03.533917    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:51:03.535025 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:03.535008    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:51:24.187049 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:51:24.187016    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:51:28.821891 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:28.821862    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-474bl"
Oct 30 02:51:28.887506 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:28.887476    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"]
Oct 30 02:51:28.887705 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:28.887684    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn" podUID="369273db-efc8-4d7c-80ed-828877e8a6e5" containerName="manager" containerID="cri-o://566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e" gracePeriod=30
Oct 30 02:51:29.133552 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.133528    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:51:29.258946 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.258914    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369273db-efc8-4d7c-80ed-828877e8a6e5-kube-api-access-6pj9d\" (UniqueName: \"kubernetes.io/projected/369273db-efc8-4d7c-80ed-828877e8a6e5-kube-api-access-6pj9d\") pod \"369273db-efc8-4d7c-80ed-828877e8a6e5\" (UID: \"369273db-efc8-4d7c-80ed-828877e8a6e5\") "
Oct 30 02:51:29.259130 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.259057    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369273db-efc8-4d7c-80ed-828877e8a6e5-cert\" (UniqueName: \"kubernetes.io/secret/369273db-efc8-4d7c-80ed-828877e8a6e5-cert\") pod \"369273db-efc8-4d7c-80ed-828877e8a6e5\" (UID: \"369273db-efc8-4d7c-80ed-828877e8a6e5\") "
Oct 30 02:51:29.261077 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.261050    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369273db-efc8-4d7c-80ed-828877e8a6e5-cert" pod "369273db-efc8-4d7c-80ed-828877e8a6e5" (UID: "369273db-efc8-4d7c-80ed-828877e8a6e5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:51:29.261200 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.261101    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369273db-efc8-4d7c-80ed-828877e8a6e5-kube-api-access-6pj9d" pod "369273db-efc8-4d7c-80ed-828877e8a6e5" (UID: "369273db-efc8-4d7c-80ed-828877e8a6e5"). InnerVolumeSpecName "kube-api-access-6pj9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:51:29.360073 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.360048    4278 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/369273db-efc8-4d7c-80ed-828877e8a6e5-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:51:29.360073 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.360072    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pj9d\" (UniqueName: \"kubernetes.io/projected/369273db-efc8-4d7c-80ed-828877e8a6e5-kube-api-access-6pj9d\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:51:29.959919 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.959888    4278 generic.go:356] "Generic (PLEG): container finished" podID="369273db-efc8-4d7c-80ed-828877e8a6e5" containerID="566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e" exitCode=0
Oct 30 02:51:29.960343 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.959937    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"
Oct 30 02:51:29.960343 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.959950    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn" event={"ID":"369273db-efc8-4d7c-80ed-828877e8a6e5","Type":"ContainerDied","Data":"566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e"}
Oct 30 02:51:29.960343 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.959991    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn" event={"ID":"369273db-efc8-4d7c-80ed-828877e8a6e5","Type":"ContainerDied","Data":"34bad282f6d4a5ac2e4bb96a902a484decc0b40eb18d023fee60c1d3fb3efffd"}
Oct 30 02:51:29.960343 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.960010    4278 scope.go:122] "RemoveContainer" containerID="566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e"
Oct 30 02:51:29.968736 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.968717    4278 scope.go:122] "RemoveContainer" containerID="566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e"
Oct 30 02:51:29.968996 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:51:29.968976    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e\": container with ID starting with 566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e not found: ID does not exist" containerID="566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e"
Oct 30 02:51:29.969028 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:29.969005    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e"} err="failed to get container status \"566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e\": rpc error: code = NotFound desc = could not find container \"566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e\": container with ID starting with 566ac6c71f793aef5ebefefac347c084ac273de744043378c4d7b7b50f3d896e not found: ID does not exist"
Oct 30 02:51:30.014626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:30.014596    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"]
Oct 30 02:51:30.027754 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:30.027734    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z54fn"]
Oct 30 02:51:30.198528 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:30.198497    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369273db-efc8-4d7c-80ed-828877e8a6e5" path="/var/lib/kubelet/pods/369273db-efc8-4d7c-80ed-828877e8a6e5/volumes"
Oct 30 02:51:50.437617 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:50.437587    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:51:54.187789 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:51:54.187753    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:51:54.199109 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:54.199081    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:51:58.005584 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:51:58.005551    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:52:02.818302 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:02.818267    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:52:07.667997 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:07.667964    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:52:24.187120 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:52:24.187070    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:52:42.038096 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.038063    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"]
Oct 30 02:52:42.038595 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.038501    4278 cpu_manager.go:414] "containerMap: removing container" podUID="369273db-efc8-4d7c-80ed-828877e8a6e5" containerName="manager"
Oct 30 02:52:42.038595 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.038514    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369273db-efc8-4d7c-80ed-828877e8a6e5" containerName="manager"
Oct 30 02:52:42.038595 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.038583    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="369273db-efc8-4d7c-80ed-828877e8a6e5" containerName="manager"
Oct 30 02:52:42.041708 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.041692    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.097159 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.097124    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"]
Oct 30 02:52:42.109076 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.109038    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.109249 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.109107    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.109249 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.109154    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.109344 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.109260    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.109344 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.109305    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.109344 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.109323    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.109472 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.109356    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz5bh\" (UniqueName: \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-kube-api-access-jz5bh\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.109538 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.109519    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.109605 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.109591    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.210738 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.210707    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.210738 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.210741    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.210980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.210771    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz5bh\" (UniqueName: \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-kube-api-access-jz5bh\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.210980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.210796    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.210980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.210846    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.210980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.210875    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.210980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.210917    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.210980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.210941    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.210980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.210973    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.211317 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.211176    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.211371 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.211332    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.211467 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.211398    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.211593 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.211572    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.211646 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.211591    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.213141 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.213123    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.213389 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.213373    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.236265 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.236244    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.236555 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.236539    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz5bh\" (UniqueName: \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-kube-api-access-jz5bh\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fp9dv\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.355240 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.355161    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:42.664376 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.664352    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"]
Oct 30 02:52:42.666182 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:52:42.666159    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6760b8c9_1381_4a45_91a7_3c8b4d67b315.slice/crio-1de74c1f0996329062128ad7986c66f3ab754a2a1325c2ad321493b9d0f64c56 WatchSource:0}: Error finding container 1de74c1f0996329062128ad7986c66f3ab754a2a1325c2ad321493b9d0f64c56: Status 404 returned error can't find the container with id 1de74c1f0996329062128ad7986c66f3ab754a2a1325c2ad321493b9d0f64c56
Oct 30 02:52:42.668276 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.668239    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:52:42.668377 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.668321    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:52:42.668377 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:42.668362    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:52:43.289840 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:43.289803    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" event={"ID":"6760b8c9-1381-4a45-91a7-3c8b4d67b315","Type":"ContainerStarted","Data":"74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c"}
Oct 30 02:52:43.290312 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:43.289953    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" event={"ID":"6760b8c9-1381-4a45-91a7-3c8b4d67b315","Type":"ContainerStarted","Data":"1de74c1f0996329062128ad7986c66f3ab754a2a1325c2ad321493b9d0f64c56"}
Oct 30 02:52:43.353262 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:43.353216    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" podStartSLOduration=1.353202412 podStartE2EDuration="1.353202412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:43.351523046 +0000 UTC m=+701.275731234" watchObservedRunningTime="2026-10-30 02:52:43.353202412 +0000 UTC m=+701.277410599"
Oct 30 02:52:43.355555 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:43.355532    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:43.356959 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:43.356936    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 02:52:43.357047 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:43.356981    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 02:52:44.356475 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:44.356445    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 02:52:44.356848 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:44.356494    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 02:52:45.355487 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:45.355449    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 02:52:45.355679 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:45.355505    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 02:52:46.355951 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:46.355917    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 02:52:46.356327 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:46.355967    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 02:52:47.356296 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:47.356254    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 02:52:47.356697 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:47.356320    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 02:52:48.356552 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:48.356517    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 02:52:48.356935 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:48.356573    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 02:52:49.355732 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:49.355692    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 02:52:49.355906 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:49.355756    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 02:52:50.618456 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:50.618391    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:50.618992 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:50.618967    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:50.619621 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:52:50.619598    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 02:52:54.187813 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:52:54.187780    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:53:23.546841 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:23.546802    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:53:24.188335 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:53:24.188299    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:53:28.305623 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.305584    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:53:28.592355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.592284    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"]
Oct 30 02:53:28.596143 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.596116    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 02:53:28.629253 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.629223    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"]
Oct 30 02:53:28.668365 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.668337    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-aeff1d\""
Oct 30 02:53:28.669896 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.669880    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-aeff1d-dockercfg-5fzvm\""
Oct 30 02:53:28.743299 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.743262    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n2hb\" (UniqueName: \"kubernetes.io/projected/c749bbed-b584-49e5-9bcb-2b81c8e32fee-kube-api-access-2n2hb\") pod \"payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7\" (UID: \"c749bbed-b584-49e5-9bcb-2b81c8e32fee\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 02:53:28.743525 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.743352    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c749bbed-b584-49e5-9bcb-2b81c8e32fee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7\" (UID: \"c749bbed-b584-49e5-9bcb-2b81c8e32fee\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 02:53:28.844734 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.844654    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n2hb\" (UniqueName: \"kubernetes.io/projected/c749bbed-b584-49e5-9bcb-2b81c8e32fee-kube-api-access-2n2hb\") pod \"payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7\" (UID: \"c749bbed-b584-49e5-9bcb-2b81c8e32fee\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 02:53:28.844734 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.844707    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c749bbed-b584-49e5-9bcb-2b81c8e32fee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7\" (UID: \"c749bbed-b584-49e5-9bcb-2b81c8e32fee\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 02:53:28.845402 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.845381    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c749bbed-b584-49e5-9bcb-2b81c8e32fee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7\" (UID: \"c749bbed-b584-49e5-9bcb-2b81c8e32fee\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 02:53:28.872368 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.872342    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n2hb\" (UniqueName: \"kubernetes.io/projected/c749bbed-b584-49e5-9bcb-2b81c8e32fee-kube-api-access-2n2hb\") pod \"payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7\" (UID: \"c749bbed-b584-49e5-9bcb-2b81c8e32fee\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 02:53:28.914493 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:28.914466    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 02:53:29.213505 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.213481    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"]
Oct 30 02:53:29.216170 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:53:29.216140    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc749bbed_b584_49e5_9bcb_2b81c8e32fee.slice/crio-fd96ac65f50273360e46e365293b366e9dec89623b1dd482e7a0815d2af62801 WatchSource:0}: Error finding container fd96ac65f50273360e46e365293b366e9dec89623b1dd482e7a0815d2af62801: Status 404 returned error can't find the container with id fd96ac65f50273360e46e365293b366e9dec89623b1dd482e7a0815d2af62801
Oct 30 02:53:29.217949 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.217929    4278 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:53:29.496168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.496087    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7" event={"ID":"c749bbed-b584-49e5-9bcb-2b81c8e32fee","Type":"ContainerStarted","Data":"fd96ac65f50273360e46e365293b366e9dec89623b1dd482e7a0815d2af62801"}
Oct 30 02:53:29.752639 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.752559    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"]
Oct 30 02:53:29.756649 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.756626    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 02:53:29.779280 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.779224    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"]
Oct 30 02:53:29.781312 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.781286    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-7d1788\""
Oct 30 02:53:29.781490 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.781475    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-7d1788-dockercfg-ct5td\""
Oct 30 02:53:29.857107 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.857067    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16051f97-ead7-430d-bbea-5da121098a02-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78\" (UID: \"16051f97-ead7-430d-bbea-5da121098a02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 02:53:29.857287 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.857176    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgz9r\" (UniqueName: \"kubernetes.io/projected/16051f97-ead7-430d-bbea-5da121098a02-kube-api-access-vgz9r\") pod \"payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78\" (UID: \"16051f97-ead7-430d-bbea-5da121098a02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 02:53:29.958595 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.958562    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16051f97-ead7-430d-bbea-5da121098a02-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78\" (UID: \"16051f97-ead7-430d-bbea-5da121098a02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 02:53:29.958782 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.958710    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgz9r\" (UniqueName: \"kubernetes.io/projected/16051f97-ead7-430d-bbea-5da121098a02-kube-api-access-vgz9r\") pod \"payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78\" (UID: \"16051f97-ead7-430d-bbea-5da121098a02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 02:53:29.959278 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.959261    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16051f97-ead7-430d-bbea-5da121098a02-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78\" (UID: \"16051f97-ead7-430d-bbea-5da121098a02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 02:53:29.980246 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:29.980226    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgz9r\" (UniqueName: \"kubernetes.io/projected/16051f97-ead7-430d-bbea-5da121098a02-kube-api-access-vgz9r\") pod \"payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78\" (UID: \"16051f97-ead7-430d-bbea-5da121098a02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 02:53:30.071590 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:30.071557    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 02:53:30.383805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:30.383676    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"]
Oct 30 02:53:30.386392 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:53:30.386358    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16051f97_ead7_430d_bbea_5da121098a02.slice/crio-e428e1c8cd14fff71f99252fcf6d340ade751ac61bd70e7af94b6e6df6e8b6e5 WatchSource:0}: Error finding container e428e1c8cd14fff71f99252fcf6d340ade751ac61bd70e7af94b6e6df6e8b6e5: Status 404 returned error can't find the container with id e428e1c8cd14fff71f99252fcf6d340ade751ac61bd70e7af94b6e6df6e8b6e5
Oct 30 02:53:30.502801 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:30.502760    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78" event={"ID":"16051f97-ead7-430d-bbea-5da121098a02","Type":"ContainerStarted","Data":"e428e1c8cd14fff71f99252fcf6d340ade751ac61bd70e7af94b6e6df6e8b6e5"}
Oct 30 02:53:31.509489 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:31.509458    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7" event={"ID":"c749bbed-b584-49e5-9bcb-2b81c8e32fee","Type":"ContainerStarted","Data":"f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4"}
Oct 30 02:53:31.509833 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:31.509564    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 02:53:31.510926 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:31.510908    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78" event={"ID":"16051f97-ead7-430d-bbea-5da121098a02","Type":"ContainerStarted","Data":"ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d"}
Oct 30 02:53:31.511038 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:31.511024    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 02:53:31.572895 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:31.572780    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7" podStartSLOduration=1.498458239 podStartE2EDuration="3.572760856s" totalImagesPullingTime="2.074302617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:31.570804657 +0000 UTC m=+749.495012843" watchObservedRunningTime="2026-10-30 02:53:31.572760856 +0000 UTC m=+749.496969126"
Oct 30 02:53:31.665010 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:31.664962    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78" podStartSLOduration=1.6141479539999999 podStartE2EDuration="2.664947411s" totalImagesPullingTime="1.050799457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:31.662813888 +0000 UTC m=+749.587022073" watchObservedRunningTime="2026-10-30 02:53:31.664947411 +0000 UTC m=+749.589155597"
Oct 30 02:53:33.083237 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:33.083203    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:53:37.519097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.519065    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"]
Oct 30 02:53:37.522743 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.522723    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.552944 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.552912    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"]
Oct 30 02:53:37.560796 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.560774    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bzc5m\""
Oct 30 02:53:37.560924 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.560852    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 02:53:37.560924 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.560897    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-810119-serving-cert\""
Oct 30 02:53:37.560983 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.560956    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 02:53:37.561027 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.561017    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 02:53:37.625296 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.625265    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-maas-api-tls\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.625476 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.625313    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cd7123-3610-4420-adaf-bb6c62de7577-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.625476 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.625448    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mmk\" (UniqueName: \"kubernetes.io/projected/75cd7123-3610-4420-adaf-bb6c62de7577-kube-api-access-n9mmk\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.625574 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.625519    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-metrics-certs\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.726533 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.726501    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-maas-api-tls\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.726533 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.726539    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cd7123-3610-4420-adaf-bb6c62de7577-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.726730 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.726563    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mmk\" (UniqueName: \"kubernetes.io/projected/75cd7123-3610-4420-adaf-bb6c62de7577-kube-api-access-n9mmk\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.726730 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.726605    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-metrics-certs\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.734871 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.727846    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cd7123-3610-4420-adaf-bb6c62de7577-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.735816 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.735535    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-maas-api-tls\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.736997 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.736972    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-metrics-certs\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.752862 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.752835    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mmk\" (UniqueName: \"kubernetes.io/projected/75cd7123-3610-4420-adaf-bb6c62de7577-kube-api-access-n9mmk\") pod \"maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:37.835879 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:37.835786    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:38.151172 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:38.151146    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"]
Oct 30 02:53:38.152307 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:53:38.152279    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cd7123_3610_4420_adaf_bb6c62de7577.slice/crio-8be42020782570642b100872fec76f495370cb3ebd6c640ff75f8817ee18a30b WatchSource:0}: Error finding container 8be42020782570642b100872fec76f495370cb3ebd6c640ff75f8817ee18a30b: Status 404 returned error can't find the container with id 8be42020782570642b100872fec76f495370cb3ebd6c640ff75f8817ee18a30b
Oct 30 02:53:38.551184 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:38.551146    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf" event={"ID":"75cd7123-3610-4420-adaf-bb6c62de7577","Type":"ContainerStarted","Data":"8be42020782570642b100872fec76f495370cb3ebd6c640ff75f8817ee18a30b"}
Oct 30 02:53:39.873669 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:39.871744    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"]
Oct 30 02:53:39.883467 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:39.883440    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:39.902754 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:39.902716    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"]
Oct 30 02:53:39.914190 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:39.913513    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"default-dockercfg-g2vqc\""
Oct 30 02:53:39.914190 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:39.913623    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"openshift-service-ca.crt\""
Oct 30 02:53:39.914190 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:39.913644    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"kube-root-ca.crt\""
Oct 30 02:53:39.915644 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:39.915613    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"simulator-w6-32bad3-kserve-self-signed-certs\""
Oct 30 02:53:40.061137 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.061099    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-dshm\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.061328 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.061167    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tmp-dir\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.061328 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.061227    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-model-cache\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.061328 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.061281    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jjp\" (UniqueName: \"kubernetes.io/projected/b4584e9e-386b-47a3-93e3-abe7a1961f4c-kube-api-access-q6jjp\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.061328 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.061305    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tls-certs\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.061543 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.061339    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-home\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.162942 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.162847    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-home\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.163123 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.162979    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-dshm\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.163123 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.163006    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tmp-dir\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.163123 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.163047    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-model-cache\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.163123 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.163109    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jjp\" (UniqueName: \"kubernetes.io/projected/b4584e9e-386b-47a3-93e3-abe7a1961f4c-kube-api-access-q6jjp\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.163280 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.163137    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tls-certs\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.164044 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.163997    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tmp-dir\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.166352 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.166324    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tls-certs\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.166604 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.166585    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-model-cache\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.169347 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.169300    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-dshm\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.169691 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.169661    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-home\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.202204 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.202169    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jjp\" (UniqueName: \"kubernetes.io/projected/b4584e9e-386b-47a3-93e3-abe7a1961f4c-kube-api-access-q6jjp\") pod \"simulator-w6-32bad3-kserve-94fd4b9d-tbzfz\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") " pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.203936 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.203914    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:40.518652 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.518631    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"]
Oct 30 02:53:40.521177 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:53:40.521154    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4584e9e_386b_47a3_93e3_abe7a1961f4c.slice/crio-768a01277591fe3ab337a184e78c234b9c96abc494da8ea4c1149da18bfd30e3 WatchSource:0}: Error finding container 768a01277591fe3ab337a184e78c234b9c96abc494da8ea4c1149da18bfd30e3: Status 404 returned error can't find the container with id 768a01277591fe3ab337a184e78c234b9c96abc494da8ea4c1149da18bfd30e3
Oct 30 02:53:40.562953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.562930    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" event={"ID":"b4584e9e-386b-47a3-93e3-abe7a1961f4c","Type":"ContainerStarted","Data":"768a01277591fe3ab337a184e78c234b9c96abc494da8ea4c1149da18bfd30e3"}
Oct 30 02:53:40.564377 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.564350    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf" event={"ID":"75cd7123-3610-4420-adaf-bb6c62de7577","Type":"ContainerStarted","Data":"b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608"}
Oct 30 02:53:40.564493 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.564446    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:40.606776 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:40.606716    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf" podStartSLOduration=1.257357196 podStartE2EDuration="3.606701566s" totalImagesPullingTime="2.34934437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:40.606349873 +0000 UTC m=+758.530558060" watchObservedRunningTime="2026-10-30 02:53:40.606701566 +0000 UTC m=+758.530909754"
Oct 30 02:53:42.527151 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:42.527121    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 02:53:42.527592 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:42.527199    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 02:53:42.577294 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:42.577260    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" event={"ID":"b4584e9e-386b-47a3-93e3-abe7a1961f4c","Type":"ContainerStarted","Data":"45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55"}
Oct 30 02:53:42.650971 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:42.650902    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podStartSLOduration=2.614630665 podStartE2EDuration="3.65088237s" totalImagesPullingTime="1.036251705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:42.649046478 +0000 UTC m=+760.573254677" watchObservedRunningTime="2026-10-30 02:53:42.65088237 +0000 UTC m=+760.575090558"
Oct 30 02:53:43.204521 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:43.204487    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:44.299379 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:44.299337    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:53:46.577259 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:46.577230    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 02:53:50.204295 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:50.204261    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:50.216366 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:50.216344    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:50.628601 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:50.628576    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:53:51.114576 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.114537    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"]
Oct 30 02:53:51.118795 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.118777    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.149963 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.149923    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-aeff1d\"/\"default-dockercfg-gqwq9\""
Oct 30 02:53:51.149963 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.149933    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-aeff1d\"/\"kube-root-ca.crt\""
Oct 30 02:53:51.150140 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.149974    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-aeff1d\"/\"openshift-service-ca.crt\""
Oct 30 02:53:51.150140 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.150032    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-aeff1d\"/\"simulator-w0-aeff1d-kserve-self-signed-certs\""
Oct 30 02:53:51.151931 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.151910    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"]
Oct 30 02:53:51.168781 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.168761    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-dshm\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.168878 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.168799    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-model-cache\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.168878 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.168822    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzhwr\" (UniqueName: \"kubernetes.io/projected/fbef8fa2-7fb0-4197-b126-3441d7420772-kube-api-access-xzhwr\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.168973 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.168898    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbef8fa2-7fb0-4197-b126-3441d7420772-tls-certs\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.169083 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.169053    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-tmp-dir\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.169135 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.169099    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-home\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.270567 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.270526    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-tmp-dir\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.271048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.270617    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-home\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.271048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.270696    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-dshm\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.271048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.270804    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-model-cache\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.271048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.270850    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzhwr\" (UniqueName: \"kubernetes.io/projected/fbef8fa2-7fb0-4197-b126-3441d7420772-kube-api-access-xzhwr\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.271048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.270903    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbef8fa2-7fb0-4197-b126-3441d7420772-tls-certs\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.271048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.271008    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-tmp-dir\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.271897 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.271638    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-home\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.271897 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.271724    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-model-cache\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.273496 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.273472    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbef8fa2-7fb0-4197-b126-3441d7420772-tls-certs\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.273861 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.273840    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-dshm\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.291147 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.291120    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzhwr\" (UniqueName: \"kubernetes.io/projected/fbef8fa2-7fb0-4197-b126-3441d7420772-kube-api-access-xzhwr\") pod \"simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") " pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.430221 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.430136    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:51.728018 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:51.727994    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"]
Oct 30 02:53:51.729605 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:53:51.729577    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbef8fa2_7fb0_4197_b126_3441d7420772.slice/crio-dc5187011aef18060f8330f880c2c1bb87f03578d630d99fe76763943e913765 WatchSource:0}: Error finding container dc5187011aef18060f8330f880c2c1bb87f03578d630d99fe76763943e913765: Status 404 returned error can't find the container with id dc5187011aef18060f8330f880c2c1bb87f03578d630d99fe76763943e913765
Oct 30 02:53:52.629805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:52.629753    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" event={"ID":"fbef8fa2-7fb0-4197-b126-3441d7420772","Type":"ContainerStarted","Data":"db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab"}
Oct 30 02:53:52.629805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:52.629805    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" event={"ID":"fbef8fa2-7fb0-4197-b126-3441d7420772","Type":"ContainerStarted","Data":"dc5187011aef18060f8330f880c2c1bb87f03578d630d99fe76763943e913765"}
Oct 30 02:53:52.670882 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:52.670826    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podStartSLOduration=1.670809582 podStartE2EDuration="1.670809582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:52.669930888 +0000 UTC m=+770.594139086" watchObservedRunningTime="2026-10-30 02:53:52.670809582 +0000 UTC m=+770.595017770"
Oct 30 02:53:53.430470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:53.430403    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:53:54.189694 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:53:54.189659    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:53:54.371985 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.371951    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"]
Oct 30 02:53:54.375568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.375549    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.396521 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.396497    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"]
Oct 30 02:53:54.401504 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.401467    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnplx\" (UniqueName: \"kubernetes.io/projected/a3ded0f0-18c0-4997-82b3-84111989074e-kube-api-access-nnplx\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.401598 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.401511    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-home\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.401598 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.401535    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-dshm\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.401681 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.401659    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ded0f0-18c0-4997-82b3-84111989074e-tls-certs\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.401824 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.401804    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-model-cache\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.401887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.401876    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-tmp-dir\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.407194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.407175    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7d1788\"/\"kube-root-ca.crt\""
Oct 30 02:53:54.407968 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.407950    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7d1788\"/\"openshift-service-ca.crt\""
Oct 30 02:53:54.408042 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.407950    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d1788\"/\"default-dockercfg-4qgt4\""
Oct 30 02:53:54.408458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.408439    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d1788\"/\"simulator-w4-7d1788-kserve-self-signed-certs\""
Oct 30 02:53:54.502752 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.502654    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-model-cache\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.502752 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.502705    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-tmp-dir\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.502752 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.502748    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnplx\" (UniqueName: \"kubernetes.io/projected/a3ded0f0-18c0-4997-82b3-84111989074e-kube-api-access-nnplx\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.503029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.502789    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-home\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.503029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.502818    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-dshm\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.503135 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.503092    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-model-cache\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.503197 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.503138    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ded0f0-18c0-4997-82b3-84111989074e-tls-certs\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.503197 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.503157    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-home\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.503440 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.503406    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-tmp-dir\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.505202 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.505173    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-dshm\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.505518 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.505497    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ded0f0-18c0-4997-82b3-84111989074e-tls-certs\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.528842 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.528808    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnplx\" (UniqueName: \"kubernetes.io/projected/a3ded0f0-18c0-4997-82b3-84111989074e-kube-api-access-nnplx\") pod \"simulator-w4-7d1788-kserve-58bc557594-zlwfl\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") " pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.689587 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.689554    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:54.989537 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:54.989497    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"]
Oct 30 02:53:54.992141 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:53:54.992115    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3ded0f0_18c0_4997_82b3_84111989074e.slice/crio-36410c4dde804eeada623062d693fe570955814c0a77c90d7d3234cfaf466ec4 WatchSource:0}: Error finding container 36410c4dde804eeada623062d693fe570955814c0a77c90d7d3234cfaf466ec4: Status 404 returned error can't find the container with id 36410c4dde804eeada623062d693fe570955814c0a77c90d7d3234cfaf466ec4
Oct 30 02:53:55.018912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:55.018884    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:53:55.648008 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:55.647972    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" event={"ID":"a3ded0f0-18c0-4997-82b3-84111989074e","Type":"ContainerStarted","Data":"90b34c0b60f6b8fd501f13c3dd59cc01b8d8a0e85e7a3aa6a00d9240e9e3ffc1"}
Oct 30 02:53:55.648372 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:55.648018    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" event={"ID":"a3ded0f0-18c0-4997-82b3-84111989074e","Type":"ContainerStarted","Data":"36410c4dde804eeada623062d693fe570955814c0a77c90d7d3234cfaf466ec4"}
Oct 30 02:53:55.690167 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:55.690075    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:53:55.691977 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:53:55.691933    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podStartSLOduration=1.691919536 podStartE2EDuration="1.691919536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:55.690630368 +0000 UTC m=+773.614838555" watchObservedRunningTime="2026-10-30 02:53:55.691919536 +0000 UTC m=+773.616127723"
Oct 30 02:54:01.430461 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:01.430407    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:54:01.442624 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:01.442603    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:54:01.686685 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:01.686612    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:54:04.689744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:04.689716    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:54:04.705237 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:04.705209    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:54:04.878054 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:04.878023    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:54:05.245670 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.245638    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"]
Oct 30 02:54:05.253184 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.253161    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.268987 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.268962    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"]
Oct 30 02:54:05.271789 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.271767    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"kube-root-ca.crt\""
Oct 30 02:54:05.271913 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.271854    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"openshift-service-ca.crt\""
Oct 30 02:54:05.271981 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.271953    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"simulator-w3-d9f3ac-kserve-self-signed-certs\""
Oct 30 02:54:05.272115 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.272097    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"default-dockercfg-4fftw\""
Oct 30 02:54:05.310110 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.310083    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-model-cache\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.310272 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.310153    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww864\" (UniqueName: \"kubernetes.io/projected/6c2f9269-eb24-4bb1-b784-da668f2e768f-kube-api-access-ww864\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.310272 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.310236    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2f9269-eb24-4bb1-b784-da668f2e768f-tls-certs\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.310371 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.310290    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-home\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.310499 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.310461    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-dshm\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.310556 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.310533    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-tmp-dir\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.412132 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.412092    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww864\" (UniqueName: \"kubernetes.io/projected/6c2f9269-eb24-4bb1-b784-da668f2e768f-kube-api-access-ww864\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.412316 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.412154    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2f9269-eb24-4bb1-b784-da668f2e768f-tls-certs\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.412316 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.412201    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-home\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.412316 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.412239    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-dshm\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.412316 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.412275    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-tmp-dir\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.412568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.412336    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-model-cache\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.412691 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.412665    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-home\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.412847 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.412822    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-tmp-dir\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.413055 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.413029    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-model-cache\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.414948 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.414924    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-dshm\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.415624 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.415599    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2f9269-eb24-4bb1-b784-da668f2e768f-tls-certs\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.441822 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.441792    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww864\" (UniqueName: \"kubernetes.io/projected/6c2f9269-eb24-4bb1-b784-da668f2e768f-kube-api-access-ww864\") pod \"simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") " pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.565666 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.565632    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:05.701608 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.701584    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:54:05.863244 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:05.863216    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"]
Oct 30 02:54:05.869604 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:54:05.869563    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c2f9269_eb24_4bb1_b784_da668f2e768f.slice/crio-8e3d456a604b358f1312a7dcb186ad2316486cec5c805a326e0918a4a9f5aec3 WatchSource:0}: Error finding container 8e3d456a604b358f1312a7dcb186ad2316486cec5c805a326e0918a4a9f5aec3: Status 404 returned error can't find the container with id 8e3d456a604b358f1312a7dcb186ad2316486cec5c805a326e0918a4a9f5aec3
Oct 30 02:54:06.708764 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:06.708728    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" event={"ID":"6c2f9269-eb24-4bb1-b784-da668f2e768f","Type":"ContainerStarted","Data":"561e1cc7b9ebbd1c89ce4d20c98ebdd374d3a91f668bef511159abf9c2dfdf5e"}
Oct 30 02:54:06.709214 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:06.708779    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" event={"ID":"6c2f9269-eb24-4bb1-b784-da668f2e768f","Type":"ContainerStarted","Data":"8e3d456a604b358f1312a7dcb186ad2316486cec5c805a326e0918a4a9f5aec3"}
Oct 30 02:54:06.750764 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:06.750715    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podStartSLOduration=1.750697621 podStartE2EDuration="1.750697621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:06.750177775 +0000 UTC m=+784.674385964" watchObservedRunningTime="2026-10-30 02:54:06.750697621 +0000 UTC m=+784.674905808"
Oct 30 02:54:07.566452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:07.566410    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:09.833476 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.833443    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"]
Oct 30 02:54:09.837679 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.837653    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:09.879017 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.878985    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f7d1a1\"/\"kube-root-ca.crt\""
Oct 30 02:54:09.879017 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.879009    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f7d1a1\"/\"default-dockercfg-b9gn5\""
Oct 30 02:54:09.879215 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.878996    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f7d1a1\"/\"openshift-service-ca.crt\""
Oct 30 02:54:09.879215 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.879102    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f7d1a1\"/\"simulator-w5-f7d1a1-kserve-self-signed-certs\""
Oct 30 02:54:09.881014 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.880991    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"]
Oct 30 02:54:09.965920 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.965886    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqdn\" (UniqueName: \"kubernetes.io/projected/34c5ad60-3ec3-4367-a4ba-304411c1595f-kube-api-access-jvqdn\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:09.966097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.965942    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-model-cache\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:09.966097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.966043    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-tmp-dir\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:09.966097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.966075    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-dshm\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:09.966269 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.966100    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-home\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:09.966269 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:09.966161    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34c5ad60-3ec3-4367-a4ba-304411c1595f-tls-certs\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.067168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.067119    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-dshm\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.067369 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.067188    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-home\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.067369 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.067230    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34c5ad60-3ec3-4367-a4ba-304411c1595f-tls-certs\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.067369 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.067297    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqdn\" (UniqueName: \"kubernetes.io/projected/34c5ad60-3ec3-4367-a4ba-304411c1595f-kube-api-access-jvqdn\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.067579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.067374    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-model-cache\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.067579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.067460    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-tmp-dir\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.067718 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.067692    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-home\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.067906 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.067806    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-tmp-dir\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.067906 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.067843    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-model-cache\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.069914 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.069889    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-dshm\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.070450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.070400    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34c5ad60-3ec3-4367-a4ba-304411c1595f-tls-certs\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.087968 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.087900    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqdn\" (UniqueName: \"kubernetes.io/projected/34c5ad60-3ec3-4367-a4ba-304411c1595f-kube-api-access-jvqdn\") pod \"simulator-w5-f7d1a1-kserve-865766759d-f9nd9\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") " pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.176921 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.176885    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:10.474474 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.474449    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"]
Oct 30 02:54:10.476106 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:54:10.476079    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34c5ad60_3ec3_4367_a4ba_304411c1595f.slice/crio-853e4fb4a2f548d05184b1870be022c9432fb1a1bfcc80407f38ff9997efa101 WatchSource:0}: Error finding container 853e4fb4a2f548d05184b1870be022c9432fb1a1bfcc80407f38ff9997efa101: Status 404 returned error can't find the container with id 853e4fb4a2f548d05184b1870be022c9432fb1a1bfcc80407f38ff9997efa101
Oct 30 02:54:10.730142 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.730053    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" event={"ID":"34c5ad60-3ec3-4367-a4ba-304411c1595f","Type":"ContainerStarted","Data":"5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d"}
Oct 30 02:54:10.730142 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.730095    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" event={"ID":"34c5ad60-3ec3-4367-a4ba-304411c1595f","Type":"ContainerStarted","Data":"853e4fb4a2f548d05184b1870be022c9432fb1a1bfcc80407f38ff9997efa101"}
Oct 30 02:54:10.774224 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:10.774175    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podStartSLOduration=1.774161951 podStartE2EDuration="1.774161951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:10.773295059 +0000 UTC m=+788.697503246" watchObservedRunningTime="2026-10-30 02:54:10.774161951 +0000 UTC m=+788.698370140"
Oct 30 02:54:11.177186 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:11.177155    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:15.566021 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:15.565986    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:15.577926 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:15.577901    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:15.766388 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:15.766358    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:54:17.910280 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:17.910244    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:54:20.177471 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:20.177439    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:20.189308 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:20.189285    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:20.788336 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:20.788307    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:54:24.187749 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:54:24.187712    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:54:32.413691 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:32.413651    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:54:41.085050 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.085015    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"]
Oct 30 02:54:41.090792 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.090769    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.105139 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.105110    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"]
Oct 30 02:54:41.116443 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.116411    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d1788\"/\"premium-w4-7d1788-kserve-self-signed-certs\""
Oct 30 02:54:41.182484 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.182449    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-model-cache\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.182650 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.182510    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-home\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.182650 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.182627    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tmp-dir\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.182772 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.182669    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-dshm\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.182772 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.182767    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tls-certs\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.182934 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.182784    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvp9\" (UniqueName: \"kubernetes.io/projected/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-kube-api-access-tdvp9\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.284513 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.283491    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-model-cache\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.284513 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.283599    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-home\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.284513 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.283657    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tmp-dir\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.284513 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.283686    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-dshm\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.284513 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.283753    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tls-certs\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.284513 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.283777    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvp9\" (UniqueName: \"kubernetes.io/projected/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-kube-api-access-tdvp9\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.284513 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.283889    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-model-cache\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.284513 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.284130    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tmp-dir\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.284513 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.284359    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-home\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.293714 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.293681    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-dshm\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.294072 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.294047    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tls-certs\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.304645 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.304616    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvp9\" (UniqueName: \"kubernetes.io/projected/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-kube-api-access-tdvp9\") pod \"premium-w4-7d1788-kserve-79ccc9d95-p96gw\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") " pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.404547 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.404453    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:41.720605 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.720578    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"]
Oct 30 02:54:41.721777 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:54:41.721752    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd85dc7d6_1fcc_4f8a_836e_7079ab8c848e.slice/crio-d28e5a1d9071db8c505c558502fe4541c58028975f4d6925a1c1861b9fc0bafe WatchSource:0}: Error finding container d28e5a1d9071db8c505c558502fe4541c58028975f4d6925a1c1861b9fc0bafe: Status 404 returned error can't find the container with id d28e5a1d9071db8c505c558502fe4541c58028975f4d6925a1c1861b9fc0bafe
Oct 30 02:54:41.889252 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.889218    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" event={"ID":"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e","Type":"ContainerStarted","Data":"beda273b6b62fc73fef3e98eae3f0692c98bc63dee3333c882d77d45bec2f36d"}
Oct 30 02:54:41.889405 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.889259    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" event={"ID":"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e","Type":"ContainerStarted","Data":"d28e5a1d9071db8c505c558502fe4541c58028975f4d6925a1c1861b9fc0bafe"}
Oct 30 02:54:41.931521 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:41.931473    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podStartSLOduration=0.9314606 podStartE2EDuration="931.4606ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:41.929710459 +0000 UTC m=+819.853918679" watchObservedRunningTime="2026-10-30 02:54:41.9314606 +0000 UTC m=+819.855668788"
Oct 30 02:54:42.405394 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:42.405365    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:43.906722 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:43.906687    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"]
Oct 30 02:54:43.910630 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:43.910609    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:43.930554 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:43.930534    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-aeff1d-openshift-default-dockercfg-8cpz8\""
Oct 30 02:54:43.945897 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:43.945874    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"]
Oct 30 02:54:44.009863 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.009830    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.010030 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.009875    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdz92\" (UniqueName: \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-kube-api-access-jdz92\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.010030 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.010003    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fbe77ca-1cd6-440f-a919-921fb337fca1-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.010159 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.010063    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.010159 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.010081    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.010159 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.010097    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.010322 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.010211    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.010322 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.010276    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.010399 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.010330    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.111813 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.111783    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.111988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.111833    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.111988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.111870    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.111988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.111905    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdz92\" (UniqueName: \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-kube-api-access-jdz92\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.111988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.111953    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fbe77ca-1cd6-440f-a919-921fb337fca1-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.112194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.112017    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.112194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.112045    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.112194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.112069    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.112194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.112131    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.112386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.112248    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.112386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.112302    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.112530 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.112467    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.112591 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.112557    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.112664 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.112636    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fbe77ca-1cd6-440f-a919-921fb337fca1-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.114328 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.114303    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.114660 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.114641    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.132378 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.132354    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.133105 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.133088    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdz92\" (UniqueName: \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-kube-api-access-jdz92\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.223553 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.223487    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:44.531914 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.531888    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"]
Oct 30 02:54:44.533601 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:54:44.533576    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbe77ca_1cd6_440f_a919_921fb337fca1.slice/crio-2a13f7e57688b75eb7528a15cc9cdf7750a90aff431aa76680a370dbe95aefe5 WatchSource:0}: Error finding container 2a13f7e57688b75eb7528a15cc9cdf7750a90aff431aa76680a370dbe95aefe5: Status 404 returned error can't find the container with id 2a13f7e57688b75eb7528a15cc9cdf7750a90aff431aa76680a370dbe95aefe5
Oct 30 02:54:44.535913 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.535882    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:54:44.536021 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.535958    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:54:44.536021 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.536000    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:54:44.910091 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.910004    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" event={"ID":"6fbe77ca-1cd6-440f-a919-921fb337fca1","Type":"ContainerStarted","Data":"fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38"}
Oct 30 02:54:44.910091 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.910057    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" event={"ID":"6fbe77ca-1cd6-440f-a919-921fb337fca1","Type":"ContainerStarted","Data":"2a13f7e57688b75eb7528a15cc9cdf7750a90aff431aa76680a370dbe95aefe5"}
Oct 30 02:54:44.988447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:44.986255    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" podStartSLOduration=1.986235063 podStartE2EDuration="1.986235063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:44.970247029 +0000 UTC m=+822.894455250" watchObservedRunningTime="2026-10-30 02:54:44.986235063 +0000 UTC m=+822.910443251"
Oct 30 02:54:45.223776 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:45.223695    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:45.225087 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:45.225065    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:54:45.225182 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:45.225115    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 02:54:46.224479 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.224419    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:54:46.224857 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.224492    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 02:54:46.566448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.561239    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"]
Oct 30 02:54:46.592891 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.592859    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.593418 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.593393    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"]
Oct 30 02:54:46.622821 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.622790    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"openshift-service-ca.crt\""
Oct 30 02:54:46.622821 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.622791    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"kube-root-ca.crt\""
Oct 30 02:54:46.622981 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.622850    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"default-dockercfg-28ksf\""
Oct 30 02:54:46.622981 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.622913    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"premium-w1-e11f03-kserve-self-signed-certs\""
Oct 30 02:54:46.738816 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.738780    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tls-certs\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.738816 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.738821    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-dshm\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.739061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.738918    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tmp-dir\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.739061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.738983    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-home\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.739164 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.739143    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-model-cache\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.739210 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.739193    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9hh\" (UniqueName: \"kubernetes.io/projected/48001cf3-8d2a-4d5f-b8f4-b341270c2100-kube-api-access-nf9hh\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.840106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.840022    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tmp-dir\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.840106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.840080    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-home\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.840320 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.840179    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-model-cache\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.840320 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.840230    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9hh\" (UniqueName: \"kubernetes.io/projected/48001cf3-8d2a-4d5f-b8f4-b341270c2100-kube-api-access-nf9hh\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.840320 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.840277    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tls-certs\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.840575 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.840318    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-dshm\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.840575 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.840493    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tmp-dir\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.840712 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.840679    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-home\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.840831 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.840807    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-model-cache\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.842698 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.842670    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-dshm\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.842806 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.842758    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tls-certs\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.877448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.863204    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9hh\" (UniqueName: \"kubernetes.io/projected/48001cf3-8d2a-4d5f-b8f4-b341270c2100-kube-api-access-nf9hh\") pod \"premium-w1-e11f03-kserve-659dcc74b5-v6crd\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") " pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:46.906681 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:46.906651    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:47.217091 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:47.217011    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"]
Oct 30 02:54:47.221076 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:54:47.221047    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48001cf3_8d2a_4d5f_b8f4_b341270c2100.slice/crio-987144823b38ac2fb26173f6c80edfa43338b968137db4e60f1b59f6177ee6c6 WatchSource:0}: Error finding container 987144823b38ac2fb26173f6c80edfa43338b968137db4e60f1b59f6177ee6c6: Status 404 returned error can't find the container with id 987144823b38ac2fb26173f6c80edfa43338b968137db4e60f1b59f6177ee6c6
Oct 30 02:54:47.223819 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:47.223794    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:54:47.223923 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:47.223841    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 02:54:47.928361 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:47.928327    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" event={"ID":"48001cf3-8d2a-4d5f-b8f4-b341270c2100","Type":"ContainerStarted","Data":"82d53b634f8ea244947919dd80c34587a7697e70a85a2442f6563b053112d62a"}
Oct 30 02:54:47.928843 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:47.928374    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" event={"ID":"48001cf3-8d2a-4d5f-b8f4-b341270c2100","Type":"ContainerStarted","Data":"987144823b38ac2fb26173f6c80edfa43338b968137db4e60f1b59f6177ee6c6"}
Oct 30 02:54:48.224308 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:48.224228    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:54:48.224308 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:48.224282    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 02:54:48.907324 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:48.907290    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:49.224277 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:49.224194    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:54:49.224277 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:49.224247    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 02:54:50.223767 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:50.223735    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:54:50.223938 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:50.223796    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 02:54:51.228173 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:51.228140    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:51.228638 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:51.228548    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:51.229312 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:51.229292    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 02:54:51.274203 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:51.274162    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podStartSLOduration=5.274149877 podStartE2EDuration="5.274149877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:47.983182371 +0000 UTC m=+825.907390571" watchObservedRunningTime="2026-10-30 02:54:51.274149877 +0000 UTC m=+829.198358106"
Oct 30 02:54:51.405606 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:51.405574    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:51.417856 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:51.417832    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:51.667166 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:51.667135    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:54:51.961790 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:51.961704    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:54:54.188281 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:54:54.188243    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:54:56.863927 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.863895    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"]
Oct 30 02:54:56.867808 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.867787    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:56.884076 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.884055    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:54:56.885550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.885524    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"]
Oct 30 02:54:56.896818 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.896785    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:54:56.906954 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.906937    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:56.918843 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.918823    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:56.966321 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.966300    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgzcb\" (UniqueName: \"kubernetes.io/projected/3bacea58-df0a-4ae3-8819-d709fb03645d-kube-api-access-hgzcb\") pod \"maas-api-revoke-sub-a966adec-jmfzd\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:56.966398 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.966338    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a966adec-jmfzd\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:56.966506 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.966490    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a966adec-jmfzd\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:56.989101 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:56.989075    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:54:57.068059 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.068031    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a966adec-jmfzd\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:57.068218 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.068185    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgzcb\" (UniqueName: \"kubernetes.io/projected/3bacea58-df0a-4ae3-8819-d709fb03645d-kube-api-access-hgzcb\") pod \"maas-api-revoke-sub-a966adec-jmfzd\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:57.068285 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.068233    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a966adec-jmfzd\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:57.068804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.068775    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a966adec-jmfzd\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:57.070762 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.070737    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a966adec-jmfzd\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:57.104404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.104373    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgzcb\" (UniqueName: \"kubernetes.io/projected/3bacea58-df0a-4ae3-8819-d709fb03645d-kube-api-access-hgzcb\") pod \"maas-api-revoke-sub-a966adec-jmfzd\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:57.181005 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.180933    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:54:57.266135 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.266103    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"]
Oct 30 02:54:57.270852 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.270830    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.293611 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.293585    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"default-dockercfg-98cf4\""
Oct 30 02:54:57.295930 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.295906    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"premium-w2-810119-kserve-self-signed-certs\""
Oct 30 02:54:57.296293 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.296277    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"kube-root-ca.crt\""
Oct 30 02:54:57.296631 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.296611    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"openshift-service-ca.crt\""
Oct 30 02:54:57.298873 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.298848    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"]
Oct 30 02:54:57.371120 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.371086    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-dshm\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.371292 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.371140    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-model-cache\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.371338 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.371306    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tls-certs\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.371490 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.371470    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-home\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.371586 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.371526    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tmp-dir\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.371639 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.371554    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpp2z\" (UniqueName: \"kubernetes.io/projected/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-kube-api-access-zpp2z\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.474098 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.472818    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tls-certs\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.474098 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.472920    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-home\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.474098 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.472962    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tmp-dir\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.474098 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.472990    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpp2z\" (UniqueName: \"kubernetes.io/projected/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-kube-api-access-zpp2z\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.474098 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.473076    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-dshm\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.474098 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.473126    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-model-cache\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.474098 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.473631    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-model-cache\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.476102 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.474896    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tmp-dir\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.476102 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.475155    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-home\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.476907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.476876    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tls-certs\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.484481 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.483888    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-dshm\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.487841 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.487303    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"]
Oct 30 02:54:57.498132 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.498104    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpp2z\" (UniqueName: \"kubernetes.io/projected/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-kube-api-access-zpp2z\") pod \"premium-w2-810119-kserve-6cff7b7ccd-44m5d\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") " pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.585290 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.585239    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:54:57.884612 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.884587    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"]
Oct 30 02:54:57.886388 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:54:57.886363    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1212a9d1_03d0_4cf6_ac31_f6768e9e3127.slice/crio-a6233f23cda5f044193a5198a14d86c72643b002af88f09712a71b58ba0b3fd5 WatchSource:0}: Error finding container a6233f23cda5f044193a5198a14d86c72643b002af88f09712a71b58ba0b3fd5: Status 404 returned error can't find the container with id a6233f23cda5f044193a5198a14d86c72643b002af88f09712a71b58ba0b3fd5
Oct 30 02:54:57.985085 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.985052    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd" event={"ID":"3bacea58-df0a-4ae3-8819-d709fb03645d","Type":"ContainerStarted","Data":"b9e5ce3adcabfa0c2a8d203e3aa81ef772853657ae9706ebfd8ed8570c6daa5c"}
Oct 30 02:54:57.986742 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.986713    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" event={"ID":"1212a9d1-03d0-4cf6-ac31-f6768e9e3127","Type":"ContainerStarted","Data":"0241739926de82216038889c851737df174af388f5e414c928061276b16ce157"}
Oct 30 02:54:57.986884 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:57.986751    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" event={"ID":"1212a9d1-03d0-4cf6-ac31-f6768e9e3127","Type":"ContainerStarted","Data":"a6233f23cda5f044193a5198a14d86c72643b002af88f09712a71b58ba0b3fd5"}
Oct 30 02:54:58.033420 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:58.033372    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podStartSLOduration=1.033359812 podStartE2EDuration="1.033359812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:58.030650254 +0000 UTC m=+835.954858441" watchObservedRunningTime="2026-10-30 02:54:58.033359812 +0000 UTC m=+835.957567978"
Oct 30 02:54:58.585944 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:54:58.585901    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:55:01.005830 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:01.005799    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd" event={"ID":"3bacea58-df0a-4ae3-8819-d709fb03645d","Type":"ContainerStarted","Data":"e2d46f7a93f95236073f739060ef6cb68113e55ed369691d858ede1355b8d66d"}
Oct 30 02:55:01.052333 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:01.052235    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd" podStartSLOduration=2.056681325 podStartE2EDuration="5.052219311s" totalImagesPullingTime="2.995537986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:01.050598224 +0000 UTC m=+838.974806422" watchObservedRunningTime="2026-10-30 02:55:01.052219311 +0000 UTC m=+838.976427526"
Oct 30 02:55:02.011949 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:02.011911    4278 generic.go:356] "Generic (PLEG): container finished" podID="3bacea58-df0a-4ae3-8819-d709fb03645d" containerID="e2d46f7a93f95236073f739060ef6cb68113e55ed369691d858ede1355b8d66d" exitCode=0
Oct 30 02:55:02.012344 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:02.011980    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd" event={"ID":"3bacea58-df0a-4ae3-8819-d709fb03645d","Type":"ContainerDied","Data":"e2d46f7a93f95236073f739060ef6cb68113e55ed369691d858ede1355b8d66d"}
Oct 30 02:55:03.167257 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:03.167232    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:55:03.237238 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:03.237204    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-openshift-ca\") pod \"3bacea58-df0a-4ae3-8819-d709fb03645d\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") "
Oct 30 02:55:03.237411 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:03.237377    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-xks-ca\") pod \"3bacea58-df0a-4ae3-8819-d709fb03645d\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") "
Oct 30 02:55:03.237506 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:03.237490    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bacea58-df0a-4ae3-8819-d709fb03645d-kube-api-access-hgzcb\" (UniqueName: \"kubernetes.io/projected/3bacea58-df0a-4ae3-8819-d709fb03645d-kube-api-access-hgzcb\") pod \"3bacea58-df0a-4ae3-8819-d709fb03645d\" (UID: \"3bacea58-df0a-4ae3-8819-d709fb03645d\") "
Oct 30 02:55:03.237979 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:03.237693    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-openshift-ca" pod "3bacea58-df0a-4ae3-8819-d709fb03645d" (UID: "3bacea58-df0a-4ae3-8819-d709fb03645d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:03.237979 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:03.237946    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:03.239700 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:03.239673    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-xks-ca" pod "3bacea58-df0a-4ae3-8819-d709fb03645d" (UID: "3bacea58-df0a-4ae3-8819-d709fb03645d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:03.240311 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:03.240284    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bacea58-df0a-4ae3-8819-d709fb03645d-kube-api-access-hgzcb" pod "3bacea58-df0a-4ae3-8819-d709fb03645d" (UID: "3bacea58-df0a-4ae3-8819-d709fb03645d"). InnerVolumeSpecName "kube-api-access-hgzcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:03.338967 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:03.338882    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bacea58-df0a-4ae3-8819-d709fb03645d-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:03.338967 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:03.338912    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgzcb\" (UniqueName: \"kubernetes.io/projected/3bacea58-df0a-4ae3-8819-d709fb03645d-kube-api-access-hgzcb\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:04.025006 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:04.024976    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"
Oct 30 02:55:04.025173 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:04.025008    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd" event={"ID":"3bacea58-df0a-4ae3-8819-d709fb03645d","Type":"ContainerDied","Data":"b9e5ce3adcabfa0c2a8d203e3aa81ef772853657ae9706ebfd8ed8570c6daa5c"}
Oct 30 02:55:04.025173 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:04.025054    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9e5ce3adcabfa0c2a8d203e3aa81ef772853657ae9706ebfd8ed8570c6daa5c"
Oct 30 02:55:07.586125 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:07.586089    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:55:07.598393 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:07.598369    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:55:08.056335 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:08.056306    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:55:10.268362 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.268330    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"]
Oct 30 02:55:10.268839 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.268823    4278 cpu_manager.go:414] "containerMap: removing container" podUID="3bacea58-df0a-4ae3-8819-d709fb03645d" containerName="revoke-keys"
Oct 30 02:55:10.268884 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.268840    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bacea58-df0a-4ae3-8819-d709fb03645d" containerName="revoke-keys"
Oct 30 02:55:10.268925 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.268916    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bacea58-df0a-4ae3-8819-d709fb03645d" containerName="revoke-keys"
Oct 30 02:55:10.274463 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.273945    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.292072 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.292044    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"]
Oct 30 02:55:10.303051 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.303031    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:55:10.303173 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.303155    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:55:10.418953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.418926    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tldpj\" (UniqueName: \"kubernetes.io/projected/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-kube-api-access-tldpj\") pod \"maas-api-revoke-sub-0e2206a7-jgdrt\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.419098 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.418998    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e2206a7-jgdrt\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.419098 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.419074    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e2206a7-jgdrt\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.520321 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.520240    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tldpj\" (UniqueName: \"kubernetes.io/projected/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-kube-api-access-tldpj\") pod \"maas-api-revoke-sub-0e2206a7-jgdrt\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.520481 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.520364    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e2206a7-jgdrt\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.520481 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.520406    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e2206a7-jgdrt\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.521070 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.521052    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e2206a7-jgdrt\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.522644 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.522621    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e2206a7-jgdrt\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.541098 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.541070    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tldpj\" (UniqueName: \"kubernetes.io/projected/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-kube-api-access-tldpj\") pod \"maas-api-revoke-sub-0e2206a7-jgdrt\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.615491 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.615456    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:10.910860 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:10.910825    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"]
Oct 30 02:55:10.911988 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:55:10.911959    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ed9263_f2a9_4c4c_9d15_8e786a4190ad.slice/crio-ea3cc82776d024ee48c183697ef86dd0deda5b823337035cf7acd97c81a1160b WatchSource:0}: Error finding container ea3cc82776d024ee48c183697ef86dd0deda5b823337035cf7acd97c81a1160b: Status 404 returned error can't find the container with id ea3cc82776d024ee48c183697ef86dd0deda5b823337035cf7acd97c81a1160b
Oct 30 02:55:11.062148 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:11.062115    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt" event={"ID":"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad","Type":"ContainerStarted","Data":"ea3cc82776d024ee48c183697ef86dd0deda5b823337035cf7acd97c81a1160b"}
Oct 30 02:55:12.068054 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:12.068015    4278 generic.go:356] "Generic (PLEG): container finished" podID="c2ed9263-f2a9-4c4c-9d15-8e786a4190ad" containerID="aad74f7c897b0a86d5dd2adcf891f97c6df675403b1fead7dab74450158081b1" exitCode=0
Oct 30 02:55:12.068407 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:12.068070    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt" event={"ID":"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad","Type":"ContainerDied","Data":"aad74f7c897b0a86d5dd2adcf891f97c6df675403b1fead7dab74450158081b1"}
Oct 30 02:55:13.207244 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:13.207223    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:13.348753 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:13.348671    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-kube-api-access-tldpj\" (UniqueName: \"kubernetes.io/projected/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-kube-api-access-tldpj\") pod \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") "
Oct 30 02:55:13.348911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:13.348793    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-xks-ca\") pod \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") "
Oct 30 02:55:13.348911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:13.348826    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-openshift-ca\") pod \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\" (UID: \"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad\") "
Oct 30 02:55:13.349204 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:13.349167    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-openshift-ca" pod "c2ed9263-f2a9-4c4c-9d15-8e786a4190ad" (UID: "c2ed9263-f2a9-4c4c-9d15-8e786a4190ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:13.350683 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:13.350659    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-xks-ca" pod "c2ed9263-f2a9-4c4c-9d15-8e786a4190ad" (UID: "c2ed9263-f2a9-4c4c-9d15-8e786a4190ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:13.350794 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:13.350665    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-kube-api-access-tldpj" pod "c2ed9263-f2a9-4c4c-9d15-8e786a4190ad" (UID: "c2ed9263-f2a9-4c4c-9d15-8e786a4190ad"). InnerVolumeSpecName "kube-api-access-tldpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:13.450169 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:13.450139    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tldpj\" (UniqueName: \"kubernetes.io/projected/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-kube-api-access-tldpj\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:13.450169 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:13.450165    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:13.450169 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:13.450175    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:14.085318 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:14.085287    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"
Oct 30 02:55:14.085500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:14.085319    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt" event={"ID":"c2ed9263-f2a9-4c4c-9d15-8e786a4190ad","Type":"ContainerDied","Data":"ea3cc82776d024ee48c183697ef86dd0deda5b823337035cf7acd97c81a1160b"}
Oct 30 02:55:14.085500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:14.085384    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3cc82776d024ee48c183697ef86dd0deda5b823337035cf7acd97c81a1160b"
Oct 30 02:55:16.494612 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:16.494532    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:55:23.765300 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.765265    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"]
Oct 30 02:55:23.765742 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.765727    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c2ed9263-f2a9-4c4c-9d15-8e786a4190ad" containerName="revoke-keys"
Oct 30 02:55:23.765742 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.765742    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ed9263-f2a9-4c4c-9d15-8e786a4190ad" containerName="revoke-keys"
Oct 30 02:55:23.765840 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.765831    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ed9263-f2a9-4c4c-9d15-8e786a4190ad" containerName="revoke-keys"
Oct 30 02:55:23.768924 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.768905    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:23.787454 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.787418    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"]
Oct 30 02:55:23.798074 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.798051    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:55:23.798173 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.798108    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:55:23.957130 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.957094    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhgct\" (UniqueName: \"kubernetes.io/projected/815a4d4b-13ca-4a7a-a4b2-75c32c402881-kube-api-access-qhgct\") pod \"maas-api-revoke-sub-9198fe5f-8npqn\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:23.957295 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.957140    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9198fe5f-8npqn\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:23.957295 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:23.957223    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9198fe5f-8npqn\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:24.058253 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:24.058173    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9198fe5f-8npqn\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:24.058253 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:24.058221    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9198fe5f-8npqn\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:24.058494 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:24.058325    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhgct\" (UniqueName: \"kubernetes.io/projected/815a4d4b-13ca-4a7a-a4b2-75c32c402881-kube-api-access-qhgct\") pod \"maas-api-revoke-sub-9198fe5f-8npqn\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:24.058810 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:24.058789    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9198fe5f-8npqn\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:24.060540 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:24.060520    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9198fe5f-8npqn\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:24.078027 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:24.078002    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhgct\" (UniqueName: \"kubernetes.io/projected/815a4d4b-13ca-4a7a-a4b2-75c32c402881-kube-api-access-qhgct\") pod \"maas-api-revoke-sub-9198fe5f-8npqn\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:24.081872 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:24.081852    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:24.189677 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:55:24.189646    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:55:24.393630 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:24.393599    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"]
Oct 30 02:55:25.142852 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:25.142820    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn" event={"ID":"815a4d4b-13ca-4a7a-a4b2-75c32c402881","Type":"ContainerStarted","Data":"78ff014415d76ee74def849e8651528dc58e1fb2f63a276e64d469b43456c7be"}
Oct 30 02:55:25.143240 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:25.142870    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn" event={"ID":"815a4d4b-13ca-4a7a-a4b2-75c32c402881","Type":"ContainerStarted","Data":"acd24fdf824c866703607b3b8b6765d8cfe8ca977fe3133b3b2a5905ab0b527e"}
Oct 30 02:55:25.186725 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:25.186665    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn" podStartSLOduration=2.186640669 podStartE2EDuration="2.186640669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:25.185746951 +0000 UTC m=+863.109955137" watchObservedRunningTime="2026-10-30 02:55:25.186640669 +0000 UTC m=+863.110848856"
Oct 30 02:55:26.149231 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:26.149196    4278 generic.go:356] "Generic (PLEG): container finished" podID="815a4d4b-13ca-4a7a-a4b2-75c32c402881" containerID="78ff014415d76ee74def849e8651528dc58e1fb2f63a276e64d469b43456c7be" exitCode=0
Oct 30 02:55:26.149662 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:26.149250    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn" event={"ID":"815a4d4b-13ca-4a7a-a4b2-75c32c402881","Type":"ContainerDied","Data":"78ff014415d76ee74def849e8651528dc58e1fb2f63a276e64d469b43456c7be"}
Oct 30 02:55:27.295408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.295378    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"]
Oct 30 02:55:27.300362 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.300341    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.308856 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.308835    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:27.316291 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.316269    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"]
Oct 30 02:55:27.392164 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.392137    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-xks-ca\") pod \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") "
Oct 30 02:55:27.392323 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.392175    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-openshift-ca\") pod \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") "
Oct 30 02:55:27.392323 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.392239    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815a4d4b-13ca-4a7a-a4b2-75c32c402881-kube-api-access-qhgct\" (UniqueName: \"kubernetes.io/projected/815a4d4b-13ca-4a7a-a4b2-75c32c402881-kube-api-access-qhgct\") pod \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\" (UID: \"815a4d4b-13ca-4a7a-a4b2-75c32c402881\") "
Oct 30 02:55:27.392559 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.392529    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6299da6-llhc8\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.392710 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.392601    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6299da6-llhc8\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.392710 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.392629    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-openshift-ca" pod "815a4d4b-13ca-4a7a-a4b2-75c32c402881" (UID: "815a4d4b-13ca-4a7a-a4b2-75c32c402881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:27.392843 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.392692    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmlp\" (UniqueName: \"kubernetes.io/projected/6a788c64-036f-47b7-a8ea-e415b0ee2f13-kube-api-access-2xmlp\") pod \"maas-api-revoke-sub-c6299da6-llhc8\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.392843 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.392813    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:27.394306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.394284    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-xks-ca" pod "815a4d4b-13ca-4a7a-a4b2-75c32c402881" (UID: "815a4d4b-13ca-4a7a-a4b2-75c32c402881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:27.394756 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.394741    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815a4d4b-13ca-4a7a-a4b2-75c32c402881-kube-api-access-qhgct" pod "815a4d4b-13ca-4a7a-a4b2-75c32c402881" (UID: "815a4d4b-13ca-4a7a-a4b2-75c32c402881"). InnerVolumeSpecName "kube-api-access-qhgct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:27.494159 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.494078    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6299da6-llhc8\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.494159 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.494125    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6299da6-llhc8\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.494371 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.494225    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmlp\" (UniqueName: \"kubernetes.io/projected/6a788c64-036f-47b7-a8ea-e415b0ee2f13-kube-api-access-2xmlp\") pod \"maas-api-revoke-sub-c6299da6-llhc8\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.494465 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.494391    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815a4d4b-13ca-4a7a-a4b2-75c32c402881-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:27.494465 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.494408    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhgct\" (UniqueName: \"kubernetes.io/projected/815a4d4b-13ca-4a7a-a4b2-75c32c402881-kube-api-access-qhgct\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:27.494853 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.494827    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6299da6-llhc8\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.496435 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.496409    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6299da6-llhc8\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.514659 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.514635    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmlp\" (UniqueName: \"kubernetes.io/projected/6a788c64-036f-47b7-a8ea-e415b0ee2f13-kube-api-access-2xmlp\") pod \"maas-api-revoke-sub-c6299da6-llhc8\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.617871 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.617834    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:27.927912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:27.927877    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"]
Oct 30 02:55:27.929404 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:55:27.929379    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a788c64_036f_47b7_a8ea_e415b0ee2f13.slice/crio-b5f3d613d00ea2fe85c44e441f6442313fc16c00ee82f7cbebe023b5b724e18d WatchSource:0}: Error finding container b5f3d613d00ea2fe85c44e441f6442313fc16c00ee82f7cbebe023b5b724e18d: Status 404 returned error can't find the container with id b5f3d613d00ea2fe85c44e441f6442313fc16c00ee82f7cbebe023b5b724e18d
Oct 30 02:55:28.163823 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.163783    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8" event={"ID":"6a788c64-036f-47b7-a8ea-e415b0ee2f13","Type":"ContainerStarted","Data":"b5f3d613d00ea2fe85c44e441f6442313fc16c00ee82f7cbebe023b5b724e18d"}
Oct 30 02:55:28.165608 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.165571    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn" event={"ID":"815a4d4b-13ca-4a7a-a4b2-75c32c402881","Type":"ContainerDied","Data":"acd24fdf824c866703607b3b8b6765d8cfe8ca977fe3133b3b2a5905ab0b527e"}
Oct 30 02:55:28.165740 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.165609    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"
Oct 30 02:55:28.165740 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.165632    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd24fdf824c866703607b3b8b6765d8cfe8ca977fe3133b3b2a5905ab0b527e"
Oct 30 02:55:28.842312 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.842282    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"]
Oct 30 02:55:28.842744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.842730    4278 cpu_manager.go:414] "containerMap: removing container" podUID="815a4d4b-13ca-4a7a-a4b2-75c32c402881" containerName="revoke-keys"
Oct 30 02:55:28.842786 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.842745    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815a4d4b-13ca-4a7a-a4b2-75c32c402881" containerName="revoke-keys"
Oct 30 02:55:28.842852 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.842843    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="815a4d4b-13ca-4a7a-a4b2-75c32c402881" containerName="revoke-keys"
Oct 30 02:55:28.845927 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.845911    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:28.862072 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.862037    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f7d1a1-openshift-default-dockercfg-54qbh\""
Oct 30 02:55:28.878245 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.878218    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"]
Oct 30 02:55:28.912079 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.912051    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:28.912241 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.912093    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfn8r\" (UniqueName: \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-kube-api-access-dfn8r\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:28.912316 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.912277    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:28.912367 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.912322    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:28.912408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.912387    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:28.912471 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.912460    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:28.912524 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.912512    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:28.912574 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.912538    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:28.912574 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:28.912556    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014002 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.013968    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014002 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014004    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014021    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014047    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014076    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014098    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014116    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014142    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014189    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfn8r\" (UniqueName: \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-kube-api-access-dfn8r\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014655 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014463    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014655 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014543    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014655 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014592    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014761 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014708    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.014953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.014935    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.016773 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.016748    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.016941 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.016924    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.034499 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.034466    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.034615 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.034584    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfn8r\" (UniqueName: \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-kube-api-access-dfn8r\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.172481 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.172380    4278 generic.go:356] "Generic (PLEG): container finished" podID="6a788c64-036f-47b7-a8ea-e415b0ee2f13" containerID="8246cdf9ae5b39d87700a45773baf044ac34911b9c4e5ebf2032fd64288daaf6" exitCode=0
Oct 30 02:55:29.172481 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.172465    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8" event={"ID":"6a788c64-036f-47b7-a8ea-e415b0ee2f13","Type":"ContainerDied","Data":"8246cdf9ae5b39d87700a45773baf044ac34911b9c4e5ebf2032fd64288daaf6"}
Oct 30 02:55:29.186591 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.186567    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:29.501298 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.501274    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"]
Oct 30 02:55:29.502691 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:55:29.502653    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1de8f8_8a94_4f5e_b74e_fcce4ccb243a.slice/crio-c5232c6dcc8ecbae666e75be49cfcb478e9e57ab4f950d00a880ebe84e537456 WatchSource:0}: Error finding container c5232c6dcc8ecbae666e75be49cfcb478e9e57ab4f950d00a880ebe84e537456: Status 404 returned error can't find the container with id c5232c6dcc8ecbae666e75be49cfcb478e9e57ab4f950d00a880ebe84e537456
Oct 30 02:55:29.505321 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.505287    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:29.505405 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.505359    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:29.505405 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.505389    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:29.553461 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.553434    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"]
Oct 30 02:55:29.558367 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.558345    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.582403 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.582374    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"]
Oct 30 02:55:29.582707 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.582685    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-810119-openshift-default-dockercfg-84mrx\""
Oct 30 02:55:29.622442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.620961    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.622442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.621074    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.622442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.621132    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.622442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.621189    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.622442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.621259    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36064a86-b94d-478a-aa06-bf99360bff43-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.622442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.621340    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36064a86-b94d-478a-aa06-bf99360bff43-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.622442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.621377    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.622442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.621468    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.622442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.621548    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxxk\" (UniqueName: \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-kube-api-access-ckxxk\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.722820 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.722790    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723015 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.722853    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxxk\" (UniqueName: \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-kube-api-access-ckxxk\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723015 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.722887    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723015 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.722916    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723015 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.722957    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723015 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.722987    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723312 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.723025    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36064a86-b94d-478a-aa06-bf99360bff43-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723312 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.723110    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36064a86-b94d-478a-aa06-bf99360bff43-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723312 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.723150    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723312 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.723231    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723569 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.723315    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723569 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.723394    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723569 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.723563    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.723776 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.723760    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36064a86-b94d-478a-aa06-bf99360bff43-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.725332 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.725308    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.725672 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.725653    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36064a86-b94d-478a-aa06-bf99360bff43-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.744749 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.744725    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.744908 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.744890    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxxk\" (UniqueName: \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-kube-api-access-ckxxk\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:29.872281 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:29.872245    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:30.180932 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.180851    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" event={"ID":"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a","Type":"ContainerStarted","Data":"35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8"}
Oct 30 02:55:30.180932 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.180904    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" event={"ID":"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a","Type":"ContainerStarted","Data":"c5232c6dcc8ecbae666e75be49cfcb478e9e57ab4f950d00a880ebe84e537456"}
Oct 30 02:55:30.187380 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.187346    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:30.188598 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.188576    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:55:30.188682 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.188616    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 02:55:30.204886 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:55:30.204858    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36064a86_b94d_478a_aa06_bf99360bff43.slice/crio-dc78520dc1dd557063f1d840e87d0df653f89b9e59967b3aa84584049ab087ae WatchSource:0}: Error finding container dc78520dc1dd557063f1d840e87d0df653f89b9e59967b3aa84584049ab087ae: Status 404 returned error can't find the container with id dc78520dc1dd557063f1d840e87d0df653f89b9e59967b3aa84584049ab087ae
Oct 30 02:55:30.207622 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.207592    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:30.207690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.207652    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:30.207724 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.207691    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:30.207945 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.207928    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"]
Oct 30 02:55:30.233539 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.233494    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" podStartSLOduration=2.233480373 podStartE2EDuration="2.233480373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:30.230862865 +0000 UTC m=+868.155071067" watchObservedRunningTime="2026-10-30 02:55:30.233480373 +0000 UTC m=+868.157688560"
Oct 30 02:55:30.333344 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.333317    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:30.431723 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.431643    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a788c64-036f-47b7-a8ea-e415b0ee2f13-kube-api-access-2xmlp\" (UniqueName: \"kubernetes.io/projected/6a788c64-036f-47b7-a8ea-e415b0ee2f13-kube-api-access-2xmlp\") pod \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") "
Oct 30 02:55:30.431866 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.431826    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-xks-ca\") pod \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") "
Oct 30 02:55:30.431926 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.431912    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-openshift-ca\") pod \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\" (UID: \"6a788c64-036f-47b7-a8ea-e415b0ee2f13\") "
Oct 30 02:55:30.432293 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.432264    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-openshift-ca" pod "6a788c64-036f-47b7-a8ea-e415b0ee2f13" (UID: "6a788c64-036f-47b7-a8ea-e415b0ee2f13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:30.433641 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.433611    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a788c64-036f-47b7-a8ea-e415b0ee2f13-kube-api-access-2xmlp" pod "6a788c64-036f-47b7-a8ea-e415b0ee2f13" (UID: "6a788c64-036f-47b7-a8ea-e415b0ee2f13"). InnerVolumeSpecName "kube-api-access-2xmlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:30.433775 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.433751    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-xks-ca" pod "6a788c64-036f-47b7-a8ea-e415b0ee2f13" (UID: "6a788c64-036f-47b7-a8ea-e415b0ee2f13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:30.534026 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.533912    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xmlp\" (UniqueName: \"kubernetes.io/projected/6a788c64-036f-47b7-a8ea-e415b0ee2f13-kube-api-access-2xmlp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:30.534026 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.533945    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:30.534026 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:30.533956    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a788c64-036f-47b7-a8ea-e415b0ee2f13-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:31.187615 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.187568    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:55:31.188224 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.188192    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 02:55:31.213404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.213372    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"
Oct 30 02:55:31.213579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.213522    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8" event={"ID":"6a788c64-036f-47b7-a8ea-e415b0ee2f13","Type":"ContainerDied","Data":"b5f3d613d00ea2fe85c44e441f6442313fc16c00ee82f7cbebe023b5b724e18d"}
Oct 30 02:55:31.213579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.213572    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5f3d613d00ea2fe85c44e441f6442313fc16c00ee82f7cbebe023b5b724e18d"
Oct 30 02:55:31.215622 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.215594    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" event={"ID":"36064a86-b94d-478a-aa06-bf99360bff43","Type":"ContainerStarted","Data":"028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336"}
Oct 30 02:55:31.215740 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.215645    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" event={"ID":"36064a86-b94d-478a-aa06-bf99360bff43","Type":"ContainerStarted","Data":"dc78520dc1dd557063f1d840e87d0df653f89b9e59967b3aa84584049ab087ae"}
Oct 30 02:55:31.411685 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.411506    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podStartSLOduration=2.41148758 podStartE2EDuration="2.41148758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:31.290751394 +0000 UTC m=+869.214959595" watchObservedRunningTime="2026-10-30 02:55:31.41148758 +0000 UTC m=+869.335695769"
Oct 30 02:55:31.880103 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.876783    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:31.897446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.890547    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:55:31.897446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:31.890612    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 02:55:32.199503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:32.198827    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:55:32.199503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:32.198884    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 02:55:32.878449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:32.876790    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:55:32.878449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:32.876849    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 02:55:33.199097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:33.199003    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:55:33.199396 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:33.199359    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 02:55:33.875458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:33.874818    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:55:33.875458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:33.874881    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 02:55:34.188503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:34.187824    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:55:34.188503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:34.187884    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 02:55:34.878445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:34.876816    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:55:34.878445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:34.876878    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 02:55:35.191503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.190819    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:55:35.191503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.190882    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 02:55:35.713515 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.710974    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"]
Oct 30 02:55:35.713515 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.711785    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6a788c64-036f-47b7-a8ea-e415b0ee2f13" containerName="revoke-keys"
Oct 30 02:55:35.713515 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.711806    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a788c64-036f-47b7-a8ea-e415b0ee2f13" containerName="revoke-keys"
Oct 30 02:55:35.713515 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.711920    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a788c64-036f-47b7-a8ea-e415b0ee2f13" containerName="revoke-keys"
Oct 30 02:55:35.719233 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.719209    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:35.726878 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.726848    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"]
Oct 30 02:55:35.808811 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.808779    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:55:35.809126 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.809090    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:55:35.840942 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.840912    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20dc1f04-gjjkj\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:35.841219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.841203    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvx8\" (UniqueName: \"kubernetes.io/projected/e1b8fb66-3018-4809-bb20-688f79c63ef8-kube-api-access-2gvx8\") pod \"maas-api-revoke-sub-20dc1f04-gjjkj\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:35.841333 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.841318    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20dc1f04-gjjkj\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:35.891461 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.884816    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:55:35.891461 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.884878    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 02:55:35.942253 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.942220    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20dc1f04-gjjkj\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:35.942471 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.942283    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvx8\" (UniqueName: \"kubernetes.io/projected/e1b8fb66-3018-4809-bb20-688f79c63ef8-kube-api-access-2gvx8\") pod \"maas-api-revoke-sub-20dc1f04-gjjkj\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:35.942471 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.942319    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20dc1f04-gjjkj\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:35.943084 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.943060    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20dc1f04-gjjkj\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:35.944579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.944556    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20dc1f04-gjjkj\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:35.974486 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:35.966720    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvx8\" (UniqueName: \"kubernetes.io/projected/e1b8fb66-3018-4809-bb20-688f79c63ef8-kube-api-access-2gvx8\") pod \"maas-api-revoke-sub-20dc1f04-gjjkj\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:36.036408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:36.036373    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:36.190470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:36.189869    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:55:36.190470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:36.189930    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 02:55:36.362924 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:36.362897    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"]
Oct 30 02:55:36.365205 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:55:36.365178    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b8fb66_3018_4809_bb20_688f79c63ef8.slice/crio-e9c340dc09504bad3e2717266ddec105e3d0ff72e7de5ad5f097a2ad80fe2b89 WatchSource:0}: Error finding container e9c340dc09504bad3e2717266ddec105e3d0ff72e7de5ad5f097a2ad80fe2b89: Status 404 returned error can't find the container with id e9c340dc09504bad3e2717266ddec105e3d0ff72e7de5ad5f097a2ad80fe2b89
Oct 30 02:55:37.001288 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:37.001223    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:37.001776 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:37.001754    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:37.003508 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:37.003472    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 02:55:37.191625 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:37.191587    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:37.192127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:37.192101    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:37.192669 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:37.192649    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 02:55:37.257817 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:37.257372    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj" event={"ID":"e1b8fb66-3018-4809-bb20-688f79c63ef8","Type":"ContainerStarted","Data":"f08678fea201d7c9f4a5fa20bcb111e944a9b0c7e09bedefb9617f44a74ababc"}
Oct 30 02:55:37.257817 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:37.257493    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj" event={"ID":"e1b8fb66-3018-4809-bb20-688f79c63ef8","Type":"ContainerStarted","Data":"e9c340dc09504bad3e2717266ddec105e3d0ff72e7de5ad5f097a2ad80fe2b89"}
Oct 30 02:55:37.339146 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:37.339094    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj" podStartSLOduration=2.339078055 podStartE2EDuration="2.339078055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:37.337820986 +0000 UTC m=+875.262029185" watchObservedRunningTime="2026-10-30 02:55:37.339078055 +0000 UTC m=+875.263286245"
Oct 30 02:55:38.264129 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:38.264093    4278 generic.go:356] "Generic (PLEG): container finished" podID="e1b8fb66-3018-4809-bb20-688f79c63ef8" containerID="f08678fea201d7c9f4a5fa20bcb111e944a9b0c7e09bedefb9617f44a74ababc" exitCode=0
Oct 30 02:55:38.264595 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:38.264186    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj" event={"ID":"e1b8fb66-3018-4809-bb20-688f79c63ef8","Type":"ContainerDied","Data":"f08678fea201d7c9f4a5fa20bcb111e944a9b0c7e09bedefb9617f44a74ababc"}
Oct 30 02:55:39.453835 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:39.453809    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:39.492932 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:39.492907    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1b8fb66-3018-4809-bb20-688f79c63ef8-kube-api-access-2gvx8\" (UniqueName: \"kubernetes.io/projected/e1b8fb66-3018-4809-bb20-688f79c63ef8-kube-api-access-2gvx8\") pod \"e1b8fb66-3018-4809-bb20-688f79c63ef8\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") "
Oct 30 02:55:39.493071 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:39.492977    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-openshift-ca\") pod \"e1b8fb66-3018-4809-bb20-688f79c63ef8\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") "
Oct 30 02:55:39.493071 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:39.493015    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-xks-ca\") pod \"e1b8fb66-3018-4809-bb20-688f79c63ef8\" (UID: \"e1b8fb66-3018-4809-bb20-688f79c63ef8\") "
Oct 30 02:55:39.493330 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:39.493301    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-openshift-ca" pod "e1b8fb66-3018-4809-bb20-688f79c63ef8" (UID: "e1b8fb66-3018-4809-bb20-688f79c63ef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:39.494878 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:39.494849    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-xks-ca" pod "e1b8fb66-3018-4809-bb20-688f79c63ef8" (UID: "e1b8fb66-3018-4809-bb20-688f79c63ef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:39.494978 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:39.494932    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b8fb66-3018-4809-bb20-688f79c63ef8-kube-api-access-2gvx8" pod "e1b8fb66-3018-4809-bb20-688f79c63ef8" (UID: "e1b8fb66-3018-4809-bb20-688f79c63ef8"). InnerVolumeSpecName "kube-api-access-2gvx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:39.593958 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:39.593882    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:39.593958 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:39.593911    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1b8fb66-3018-4809-bb20-688f79c63ef8-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:39.593958 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:39.593922    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gvx8\" (UniqueName: \"kubernetes.io/projected/e1b8fb66-3018-4809-bb20-688f79c63ef8-kube-api-access-2gvx8\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:40.208557 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.208530    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"]
Oct 30 02:55:40.208995 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.208983    4278 cpu_manager.go:414] "containerMap: removing container" podUID="e1b8fb66-3018-4809-bb20-688f79c63ef8" containerName="revoke-keys"
Oct 30 02:55:40.209045 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.208996    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1b8fb66-3018-4809-bb20-688f79c63ef8" containerName="revoke-keys"
Oct 30 02:55:40.209080 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.209067    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1b8fb66-3018-4809-bb20-688f79c63ef8" containerName="revoke-keys"
Oct 30 02:55:40.212351 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.212334    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.232069 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.232047    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"]
Oct 30 02:55:40.276070 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.276042    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj" event={"ID":"e1b8fb66-3018-4809-bb20-688f79c63ef8","Type":"ContainerDied","Data":"e9c340dc09504bad3e2717266ddec105e3d0ff72e7de5ad5f097a2ad80fe2b89"}
Oct 30 02:55:40.276070 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.276060    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"
Oct 30 02:55:40.276215 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.276089    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9c340dc09504bad3e2717266ddec105e3d0ff72e7de5ad5f097a2ad80fe2b89"
Oct 30 02:55:40.302109 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.302083    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90e78d35-thq59\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.302215 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.302117    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90e78d35-thq59\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.302309 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.302289    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6j4\" (UniqueName: \"kubernetes.io/projected/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-kube-api-access-jw6j4\") pod \"maas-api-revoke-sub-90e78d35-thq59\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.403590 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.403562    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6j4\" (UniqueName: \"kubernetes.io/projected/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-kube-api-access-jw6j4\") pod \"maas-api-revoke-sub-90e78d35-thq59\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.403710 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.403602    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90e78d35-thq59\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.403710 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.403621    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90e78d35-thq59\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.404202 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.404179    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90e78d35-thq59\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.405891 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.405869    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90e78d35-thq59\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.423931 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.423912    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6j4\" (UniqueName: \"kubernetes.io/projected/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-kube-api-access-jw6j4\") pod \"maas-api-revoke-sub-90e78d35-thq59\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.525923 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.525893    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:40.824168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:40.824142    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"]
Oct 30 02:55:40.825587 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:55:40.825564    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8417cff_2dcc_44c2_bd63_edaa5a4b6679.slice/crio-1172244ddf21586f132a12506b0ae6079db757c73cdb383aae830bbcfd4ed733 WatchSource:0}: Error finding container 1172244ddf21586f132a12506b0ae6079db757c73cdb383aae830bbcfd4ed733: Status 404 returned error can't find the container with id 1172244ddf21586f132a12506b0ae6079db757c73cdb383aae830bbcfd4ed733
Oct 30 02:55:41.282219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:41.282185    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59" event={"ID":"c8417cff-2dcc-44c2-bd63-edaa5a4b6679","Type":"ContainerStarted","Data":"ffe3aabae8697739e8dc6bb42ce95a5429476d3313d33558bd4c7b11893432be"}
Oct 30 02:55:41.282346 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:41.282268    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59" event={"ID":"c8417cff-2dcc-44c2-bd63-edaa5a4b6679","Type":"ContainerStarted","Data":"1172244ddf21586f132a12506b0ae6079db757c73cdb383aae830bbcfd4ed733"}
Oct 30 02:55:41.326726 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:41.326642    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59" podStartSLOduration=1.3266284609999999 podStartE2EDuration="1.326628461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:41.324574122 +0000 UTC m=+879.248782308" watchObservedRunningTime="2026-10-30 02:55:41.326628461 +0000 UTC m=+879.250836647"
Oct 30 02:55:42.288705 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:42.288674    4278 generic.go:356] "Generic (PLEG): container finished" podID="c8417cff-2dcc-44c2-bd63-edaa5a4b6679" containerID="ffe3aabae8697739e8dc6bb42ce95a5429476d3313d33558bd4c7b11893432be" exitCode=0
Oct 30 02:55:42.288705 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:42.288706    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59" event={"ID":"c8417cff-2dcc-44c2-bd63-edaa5a4b6679","Type":"ContainerDied","Data":"ffe3aabae8697739e8dc6bb42ce95a5429476d3313d33558bd4c7b11893432be"}
Oct 30 02:55:43.381619 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.381586    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:55:43.437007 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.436984    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:43.537204 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.537170    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-kube-api-access-jw6j4\" (UniqueName: \"kubernetes.io/projected/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-kube-api-access-jw6j4\") pod \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") "
Oct 30 02:55:43.537366 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.537342    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-xks-ca\") pod \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") "
Oct 30 02:55:43.537420 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.537376    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-openshift-ca\") pod \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\" (UID: \"c8417cff-2dcc-44c2-bd63-edaa5a4b6679\") "
Oct 30 02:55:43.537846 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.537816    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-openshift-ca" pod "c8417cff-2dcc-44c2-bd63-edaa5a4b6679" (UID: "c8417cff-2dcc-44c2-bd63-edaa5a4b6679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:43.539219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.539191    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-kube-api-access-jw6j4" pod "c8417cff-2dcc-44c2-bd63-edaa5a4b6679" (UID: "c8417cff-2dcc-44c2-bd63-edaa5a4b6679"). InnerVolumeSpecName "kube-api-access-jw6j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:43.539347 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.539327    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-xks-ca" pod "c8417cff-2dcc-44c2-bd63-edaa5a4b6679" (UID: "c8417cff-2dcc-44c2-bd63-edaa5a4b6679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:43.580456 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.580409    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"]
Oct 30 02:55:43.580880 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.580863    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c8417cff-2dcc-44c2-bd63-edaa5a4b6679" containerName="revoke-keys"
Oct 30 02:55:43.580880 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.580877    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8417cff-2dcc-44c2-bd63-edaa5a4b6679" containerName="revoke-keys"
Oct 30 02:55:43.581036 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.580963    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8417cff-2dcc-44c2-bd63-edaa5a4b6679" containerName="revoke-keys"
Oct 30 02:55:43.584304 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.584288    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.602552 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.602520    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-32bad3-openshift-default-dockercfg-wg2rd\""
Oct 30 02:55:43.605198 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.605174    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"]
Oct 30 02:55:43.638410 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638387    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tznc2\" (UniqueName: \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-kube-api-access-tznc2\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.638527 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638446    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.638527 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638511    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.638601 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638550    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.638663 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638646    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.638713 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638699    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.638747 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638732    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54319ced-9220-46fb-a97e-6c0b1e702d50-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.638776 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638763    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.638851 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638836    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.638909 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638900    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:43.638947 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638912    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:43.638947 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.638921    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6j4\" (UniqueName: \"kubernetes.io/projected/c8417cff-2dcc-44c2-bd63-edaa5a4b6679-kube-api-access-jw6j4\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:43.739800 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.739768    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.739800 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.739804    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740014 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.739829    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54319ced-9220-46fb-a97e-6c0b1e702d50-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740014 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.739857    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740014 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.739886    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740014 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.739935    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tznc2\" (UniqueName: \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-kube-api-access-tznc2\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740014 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.739971    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740014 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.740002    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740327 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.740023    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740327 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.740192    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.740473    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740620 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.740598    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54319ced-9220-46fb-a97e-6c0b1e702d50-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740682 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.740609    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.740737 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.740710    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.742084 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.742065    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.742438 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.742403    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.760174 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.760144    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.760307 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.760288    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tznc2\" (UniqueName: \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-kube-api-access-tznc2\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:43.898960 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:43.898888    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:44.218129 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:44.217528    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"]
Oct 30 02:55:44.220265 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:55:44.220240    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54319ced_9220_46fb_a97e_6c0b1e702d50.slice/crio-4186d5ba7ab30ad54430f0964b427527107859911a4dbb858e808b4637641780 WatchSource:0}: Error finding container 4186d5ba7ab30ad54430f0964b427527107859911a4dbb858e808b4637641780: Status 404 returned error can't find the container with id 4186d5ba7ab30ad54430f0964b427527107859911a4dbb858e808b4637641780
Oct 30 02:55:44.222886 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:44.222847    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:44.222963 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:44.222932    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:44.223009 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:44.222976    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:44.302285 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:44.302254    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"
Oct 30 02:55:44.302467 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:44.302252    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59" event={"ID":"c8417cff-2dcc-44c2-bd63-edaa5a4b6679","Type":"ContainerDied","Data":"1172244ddf21586f132a12506b0ae6079db757c73cdb383aae830bbcfd4ed733"}
Oct 30 02:55:44.302467 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:44.302461    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1172244ddf21586f132a12506b0ae6079db757c73cdb383aae830bbcfd4ed733"
Oct 30 02:55:44.304288 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:44.304260    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" event={"ID":"54319ced-9220-46fb-a97e-6c0b1e702d50","Type":"ContainerStarted","Data":"4186d5ba7ab30ad54430f0964b427527107859911a4dbb858e808b4637641780"}
Oct 30 02:55:45.311735 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:45.311696    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" event={"ID":"54319ced-9220-46fb-a97e-6c0b1e702d50","Type":"ContainerStarted","Data":"910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199"}
Oct 30 02:55:45.390097 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:45.390043    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" podStartSLOduration=2.390028267 podStartE2EDuration="2.390028267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:45.376037311 +0000 UTC m=+883.300245499" watchObservedRunningTime="2026-10-30 02:55:45.390028267 +0000 UTC m=+883.314236453"
Oct 30 02:55:45.899697 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:45.899663    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:45.901383 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:45.901357    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 02:55:45.901534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:45.901435    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" podUID="54319ced-9220-46fb-a97e-6c0b1e702d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 02:55:46.900017 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:46.899977    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 02:55:46.900447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:46.900035    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" podUID="54319ced-9220-46fb-a97e-6c0b1e702d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 02:55:47.899523 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:47.899485    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 02:55:47.899697 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:47.899552    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" podUID="54319ced-9220-46fb-a97e-6c0b1e702d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 02:55:48.899334 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:48.899298    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 02:55:48.899734 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:48.899357    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" podUID="54319ced-9220-46fb-a97e-6c0b1e702d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 02:55:49.899955 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:49.899917    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 02:55:49.900356 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:49.899977    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" podUID="54319ced-9220-46fb-a97e-6c0b1e702d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 02:55:50.587900 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.587866    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"]
Oct 30 02:55:50.591852 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.591831    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.612605 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.612576    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"]
Oct 30 02:55:50.620925 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.620874    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:55:50.620925 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.620905    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:55:50.644525 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.644493    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc52d0f2-9ltgb\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.644663 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.644610    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc52d0f2-9ltgb\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.644721 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.644686    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmpv8\" (UniqueName: \"kubernetes.io/projected/5b6fbe18-5093-4e5a-8545-50f22759f44f-kube-api-access-kmpv8\") pod \"maas-api-revoke-sub-cc52d0f2-9ltgb\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.745812 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.745761    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc52d0f2-9ltgb\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.746060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.745849    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc52d0f2-9ltgb\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.746060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.745901    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmpv8\" (UniqueName: \"kubernetes.io/projected/5b6fbe18-5093-4e5a-8545-50f22759f44f-kube-api-access-kmpv8\") pod \"maas-api-revoke-sub-cc52d0f2-9ltgb\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.746538 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.746509    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc52d0f2-9ltgb\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.748276 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.748243    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc52d0f2-9ltgb\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.767310 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.767284    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmpv8\" (UniqueName: \"kubernetes.io/projected/5b6fbe18-5093-4e5a-8545-50f22759f44f-kube-api-access-kmpv8\") pod \"maas-api-revoke-sub-cc52d0f2-9ltgb\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.910799 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.910709    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:50.972871 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.972840    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:50.973303 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.973279    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:50.974334 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:50.974310    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 02:55:51.216404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:51.216374    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"]
Oct 30 02:55:51.217747 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:55:51.217714    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6fbe18_5093_4e5a_8545_50f22759f44f.slice/crio-96a9e1fa3de56933c99e5ec30441efbcae000106a6aff34da945e5657ae5712b WatchSource:0}: Error finding container 96a9e1fa3de56933c99e5ec30441efbcae000106a6aff34da945e5657ae5712b: Status 404 returned error can't find the container with id 96a9e1fa3de56933c99e5ec30441efbcae000106a6aff34da945e5657ae5712b
Oct 30 02:55:51.343685 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:51.343655    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb" event={"ID":"5b6fbe18-5093-4e5a-8545-50f22759f44f","Type":"ContainerStarted","Data":"96a9e1fa3de56933c99e5ec30441efbcae000106a6aff34da945e5657ae5712b"}
Oct 30 02:55:52.351117 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:52.351084    4278 generic.go:356] "Generic (PLEG): container finished" podID="5b6fbe18-5093-4e5a-8545-50f22759f44f" containerID="1de55fdf1f041296bbd668fa13f26917b490e06c625d3c26b666ddfceb619e88" exitCode=0
Oct 30 02:55:52.351541 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:52.351173    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb" event={"ID":"5b6fbe18-5093-4e5a-8545-50f22759f44f","Type":"ContainerDied","Data":"1de55fdf1f041296bbd668fa13f26917b490e06c625d3c26b666ddfceb619e88"}
Oct 30 02:55:53.502486 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:53.502462    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:53.575680 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:53.575648    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-xks-ca\") pod \"5b6fbe18-5093-4e5a-8545-50f22759f44f\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") "
Oct 30 02:55:53.575847 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:53.575777    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-openshift-ca\") pod \"5b6fbe18-5093-4e5a-8545-50f22759f44f\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") "
Oct 30 02:55:53.575847 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:53.575820    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6fbe18-5093-4e5a-8545-50f22759f44f-kube-api-access-kmpv8\" (UniqueName: \"kubernetes.io/projected/5b6fbe18-5093-4e5a-8545-50f22759f44f-kube-api-access-kmpv8\") pod \"5b6fbe18-5093-4e5a-8545-50f22759f44f\" (UID: \"5b6fbe18-5093-4e5a-8545-50f22759f44f\") "
Oct 30 02:55:53.576186 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:53.576156    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-openshift-ca" pod "5b6fbe18-5093-4e5a-8545-50f22759f44f" (UID: "5b6fbe18-5093-4e5a-8545-50f22759f44f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:53.576288 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:53.576273    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:53.577721 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:53.577695    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-xks-ca" pod "5b6fbe18-5093-4e5a-8545-50f22759f44f" (UID: "5b6fbe18-5093-4e5a-8545-50f22759f44f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:53.577832 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:53.577810    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6fbe18-5093-4e5a-8545-50f22759f44f-kube-api-access-kmpv8" pod "5b6fbe18-5093-4e5a-8545-50f22759f44f" (UID: "5b6fbe18-5093-4e5a-8545-50f22759f44f"). InnerVolumeSpecName "kube-api-access-kmpv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:53.677374 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:53.677310    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmpv8\" (UniqueName: \"kubernetes.io/projected/5b6fbe18-5093-4e5a-8545-50f22759f44f-kube-api-access-kmpv8\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:53.677374 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:53.677335    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6fbe18-5093-4e5a-8545-50f22759f44f-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:55:54.188730 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:55:54.188695    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:55:54.365697 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:54.365669    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"
Oct 30 02:55:54.365862 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:54.365713    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb" event={"ID":"5b6fbe18-5093-4e5a-8545-50f22759f44f","Type":"ContainerDied","Data":"96a9e1fa3de56933c99e5ec30441efbcae000106a6aff34da945e5657ae5712b"}
Oct 30 02:55:54.365862 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:54.365768    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96a9e1fa3de56933c99e5ec30441efbcae000106a6aff34da945e5657ae5712b"
Oct 30 02:55:58.781533 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.781497    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"]
Oct 30 02:55:58.784075 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.781987    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5b6fbe18-5093-4e5a-8545-50f22759f44f" containerName="revoke-keys"
Oct 30 02:55:58.784075 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.782003    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6fbe18-5093-4e5a-8545-50f22759f44f" containerName="revoke-keys"
Oct 30 02:55:58.784075 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.782101    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6fbe18-5093-4e5a-8545-50f22759f44f" containerName="revoke-keys"
Oct 30 02:55:58.785249 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.785232    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:58.805568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.805538    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d9f3ac-openshift-default-dockercfg-97t95\""
Oct 30 02:55:58.831523 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.828097    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"]
Oct 30 02:55:58.936118 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.936085    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:58.936118 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.936126    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:58.936359 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.936253    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvht\" (UniqueName: \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-kube-api-access-zlvht\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:58.936418 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.936389    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:58.936493 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.936477    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:58.936550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.936496    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:58.936550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.936511    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82519a5-4fac-40c4-a8c2-230c028d2934-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:58.936653 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.936610    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:58.936653 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:58.936643    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.037646 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.037563    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.037646 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.037613    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.037869 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.037670    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.037869 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.037693    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.037869 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.037730    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvht\" (UniqueName: \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-kube-api-access-zlvht\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.037869 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.037781    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.037869 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.037830    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.037869 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.037849    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.037869 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.037871    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82519a5-4fac-40c4-a8c2-230c028d2934-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.038242 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.038042    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.038542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.038511    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.038688 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.038540    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82519a5-4fac-40c4-a8c2-230c028d2934-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.038688 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.038565    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.038688 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.038606    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.040208 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.040181    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.040322 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.040246    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.058237 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.058210    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.058335 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.058300    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvht\" (UniqueName: \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-kube-api-access-zlvht\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.101503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.101458    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:55:59.448210 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.448186    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"]
Oct 30 02:55:59.450433 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:55:59.450390    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82519a5_4fac_40c4_a8c2_230c028d2934.slice/crio-d86645d0dd4ab98f70abeabc14533fcc6fc05cc9bc3b0a1415d5a0a72bd5e6b6 WatchSource:0}: Error finding container d86645d0dd4ab98f70abeabc14533fcc6fc05cc9bc3b0a1415d5a0a72bd5e6b6: Status 404 returned error can't find the container with id d86645d0dd4ab98f70abeabc14533fcc6fc05cc9bc3b0a1415d5a0a72bd5e6b6
Oct 30 02:55:59.452904 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.452871    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:59.453000 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.452951    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:59.453059 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.453000    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:59.461843 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.461817    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"]
Oct 30 02:55:59.467891 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.467868    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.487180 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.487144    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7d1788-openshift-default-dockercfg-29c4p\""
Oct 30 02:55:59.487753 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.487692    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"]
Oct 30 02:55:59.543808 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.543774    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.543933 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.543821    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.544000 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.543928    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.544073 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.544056    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.544134 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.544111    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr492\" (UniqueName: \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-kube-api-access-vr492\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.544202 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.544184    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.544237 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.544209    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.544281 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.544243    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.544362 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.544343    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.645799 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.645769    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.645975 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.645820    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.645975 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.645866    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.645975 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.645942    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.646187 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.645987    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr492\" (UniqueName: \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-kube-api-access-vr492\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.646187 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.646035    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.646187 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.646061    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.646187 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.646119    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.646187 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.646161    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.646440 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.646242    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.646440 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.646344    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.646727 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.646707    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.646840 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.646724    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.646840 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.646751    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.648886 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.648859    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.649050 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.649024    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.667074 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.667039    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr492\" (UniqueName: \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-kube-api-access-vr492\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.667195 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.667095    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:55:59.799446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:55:59.798514    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:56:00.128298 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.128270    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"]
Oct 30 02:56:00.129334 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:56:00.129308    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71629bd_8e5e_4a49_a565_2cb10bc1aa1f.slice/crio-0ed2508b9c518aa5dfeb26655fdaeb54f964f6c90e81408a3621d836fd8777eb WatchSource:0}: Error finding container 0ed2508b9c518aa5dfeb26655fdaeb54f964f6c90e81408a3621d836fd8777eb: Status 404 returned error can't find the container with id 0ed2508b9c518aa5dfeb26655fdaeb54f964f6c90e81408a3621d836fd8777eb
Oct 30 02:56:00.131813 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.131777    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:00.131887 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.131860    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:00.131927 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.131900    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:00.401566 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.401461    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" event={"ID":"a82519a5-4fac-40c4-a8c2-230c028d2934","Type":"ContainerStarted","Data":"4b48efecd1f2208f61895e4d5d67fdc8611abbc76b8bc6416231bf994a84a01b"}
Oct 30 02:56:00.401566 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.401545    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" event={"ID":"a82519a5-4fac-40c4-a8c2-230c028d2934","Type":"ContainerStarted","Data":"d86645d0dd4ab98f70abeabc14533fcc6fc05cc9bc3b0a1415d5a0a72bd5e6b6"}
Oct 30 02:56:00.403177 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.403154    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" event={"ID":"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f","Type":"ContainerStarted","Data":"b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3"}
Oct 30 02:56:00.403304 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.403195    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" event={"ID":"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f","Type":"ContainerStarted","Data":"0ed2508b9c518aa5dfeb26655fdaeb54f964f6c90e81408a3621d836fd8777eb"}
Oct 30 02:56:00.449727 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.449682    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" podStartSLOduration=2.449669953 podStartE2EDuration="2.449669953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:00.447194166 +0000 UTC m=+898.371402353" watchObservedRunningTime="2026-10-30 02:56:00.449669953 +0000 UTC m=+898.373878139"
Oct 30 02:56:00.500416 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.498182    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" podStartSLOduration=1.498161678 podStartE2EDuration="1.498161678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:00.49504727 +0000 UTC m=+898.419255457" watchObservedRunningTime="2026-10-30 02:56:00.498161678 +0000 UTC m=+898.422369867"
Oct 30 02:56:00.799448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:00.799368    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:56:01.108507 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:01.107672    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:56:01.110447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:01.109541    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 02:56:01.110447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:01.109600    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 02:56:01.813465 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:01.809406    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 02:56:01.813465 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:01.809496    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 02:56:02.107980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:02.107883    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 02:56:02.108259 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:02.108224    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 02:56:02.808452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:02.807614    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 02:56:02.808452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:02.807689    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 02:56:03.106504 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:03.105811    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 02:56:03.106504 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:03.105874    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 02:56:03.799911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:03.799866    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 02:56:03.800646 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:03.800208    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 02:56:04.103507 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:04.102833    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 02:56:04.103507 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:04.102895    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 02:56:04.800449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:04.799790    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 02:56:04.800449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:04.799851    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 02:56:05.105517 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:05.104905    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 02:56:05.105517 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:05.104989    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 02:56:05.799776 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:05.799742    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 02:56:05.799977 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:05.799810    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 02:56:06.334910 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:06.334877    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:56:06.335970 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:06.335569    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:56:06.337016 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:06.336992    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 02:56:06.893655 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:06.893628    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:56:06.893989 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:06.893964    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:56:06.895213 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:06.895183    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 02:56:09.186711 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:09.186673    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:56:14.720877 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.720847    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"]
Oct 30 02:56:14.726528 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.726504    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.747320 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.747294    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"]
Oct 30 02:56:14.806973 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.806945    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.807117 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.807000    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.807117 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.807070    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.807191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.807126    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.807191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.807157    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.807191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.807173    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgkc\" (UniqueName: \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-kube-api-access-gqgkc\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.807285 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.807199    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.807350 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.807330    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.807481 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.807466    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.908646 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.908612    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.908646 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.908652    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.908880 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.908694    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.908880 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.908739    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.908880 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.908761    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgkc\" (UniqueName: \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-kube-api-access-gqgkc\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.908880 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.908801    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.908880 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.908844    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.909135 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.909039    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.909135 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.909088    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.909135 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.909092    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.909135 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.909113    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.909367 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.909340    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.909454 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.909361    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.909590 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.909568    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.910935 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.910913    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.911482 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.911459    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.929349 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.929321    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgkc\" (UniqueName: \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-kube-api-access-gqgkc\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:14.929760 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:14.929736    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:15.042062 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:15.042026    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:15.354449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:15.354407    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"]
Oct 30 02:56:15.356381 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:56:15.356358    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03a29e0_ae09_4eb5_aab7_0913ab7e00b0.slice/crio-ea7c819fe1ade3d47f6a0118fcb53dfbb9af1e1747f6a3a6928212a8f7f68e35 WatchSource:0}: Error finding container ea7c819fe1ade3d47f6a0118fcb53dfbb9af1e1747f6a3a6928212a8f7f68e35: Status 404 returned error can't find the container with id ea7c819fe1ade3d47f6a0118fcb53dfbb9af1e1747f6a3a6928212a8f7f68e35
Oct 30 02:56:15.358647 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:15.358617    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:15.358747 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:15.358690    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:15.358747 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:15.358735    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:15.501137 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:15.501100    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" event={"ID":"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0","Type":"ContainerStarted","Data":"5290a680e48b9c1274bc19e29b56ec2086da0bb5bb7d447c0d935f907640d734"}
Oct 30 02:56:15.501285 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:15.501157    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" event={"ID":"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0","Type":"ContainerStarted","Data":"ea7c819fe1ade3d47f6a0118fcb53dfbb9af1e1747f6a3a6928212a8f7f68e35"}
Oct 30 02:56:15.545941 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:15.545886    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podStartSLOduration=1.5458696779999999 podStartE2EDuration="1.545869678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:15.544726201 +0000 UTC m=+913.468934390" watchObservedRunningTime="2026-10-30 02:56:15.545869678 +0000 UTC m=+913.470077865"
Oct 30 02:56:16.042321 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:16.042287    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:16.043621 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:16.043600    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:56:16.043692 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:16.043645    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 02:56:17.043194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:17.043153    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:56:17.043604 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:17.043216    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 02:56:18.043250 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:18.043219    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:56:18.043689 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:18.043270    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 02:56:19.043013 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:19.042982    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:56:19.043184 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:19.043039    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 02:56:20.042976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:20.042940    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:56:20.043404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:20.042993    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 02:56:21.042876 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:21.042840    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:56:21.043045 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:21.042900    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 02:56:22.046838 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:22.046809    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:22.047281 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:22.047260    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:22.047918 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:22.047896    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 02:56:24.189200 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:56:24.189157    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:56:36.434783 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:36.434742    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:56:40.117284 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.117241    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"]
Oct 30 02:56:40.122807 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.122783    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.141287 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.141256    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"]
Oct 30 02:56:40.149541 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.149515    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-aeff1d\"/\"distinct-2-w0-aeff1d-kserve-self-signed-certs\""
Oct 30 02:56:40.250733 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.250699    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-home\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.250907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.250762    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzsjp\" (UniqueName: \"kubernetes.io/projected/fd65ad7e-0c67-473f-8199-5af294755d6c-kube-api-access-mzsjp\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.250907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.250885    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-dshm\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.250998 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.250959    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-tmp-dir\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.251034 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.250996    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd65ad7e-0c67-473f-8199-5af294755d6c-tls-certs\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.251185 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.251159    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-model-cache\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.352575 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.352535    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-home\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.352792 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.352601    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzsjp\" (UniqueName: \"kubernetes.io/projected/fd65ad7e-0c67-473f-8199-5af294755d6c-kube-api-access-mzsjp\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.352792 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.352674    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-dshm\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.352792 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.352699    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-tmp-dir\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.352792 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.352723    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd65ad7e-0c67-473f-8199-5af294755d6c-tls-certs\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.353006 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.352819    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-model-cache\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.353060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.353027    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-home\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.353268 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.353247    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-model-cache\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.353549 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.353526    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-tmp-dir\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.356034 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.355996    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-dshm\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.362566 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.362541    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd65ad7e-0c67-473f-8199-5af294755d6c-tls-certs\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.382794 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.382656    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzsjp\" (UniqueName: \"kubernetes.io/projected/fd65ad7e-0c67-473f-8199-5af294755d6c-kube-api-access-mzsjp\") pod \"distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.441800 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.441774    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:40.741300 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:40.741274    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"]
Oct 30 02:56:40.743994 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:56:40.743968    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd65ad7e_0c67_473f_8199_5af294755d6c.slice/crio-3c2da9e2f24319c34f77c0ca21e91f99d3d6f182882229a0a34df25a84b1e11a WatchSource:0}: Error finding container 3c2da9e2f24319c34f77c0ca21e91f99d3d6f182882229a0a34df25a84b1e11a: Status 404 returned error can't find the container with id 3c2da9e2f24319c34f77c0ca21e91f99d3d6f182882229a0a34df25a84b1e11a
Oct 30 02:56:41.739896 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:41.739844    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" event={"ID":"fd65ad7e-0c67-473f-8199-5af294755d6c","Type":"ContainerStarted","Data":"672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf"}
Oct 30 02:56:41.739896 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:41.739897    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" event={"ID":"fd65ad7e-0c67-473f-8199-5af294755d6c","Type":"ContainerStarted","Data":"3c2da9e2f24319c34f77c0ca21e91f99d3d6f182882229a0a34df25a84b1e11a"}
Oct 30 02:56:41.783536 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:41.783479    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podStartSLOduration=1.783461811 podStartE2EDuration="1.783461811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:41.78029423 +0000 UTC m=+939.704502416" watchObservedRunningTime="2026-10-30 02:56:41.783461811 +0000 UTC m=+939.707670002"
Oct 30 02:56:42.442588 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:42.442549    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:43.924265 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:43.924234    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"]
Oct 30 02:56:43.928262 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:43.928239    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:43.953579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:43.953551    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"]
Oct 30 02:56:44.094243 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.094212    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hkf\" (UniqueName: \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-kube-api-access-h9hkf\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.094452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.094255    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.094452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.094284    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.094452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.094409    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.094594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.094479    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.094594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.094513    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.094594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.094577    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.094706 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.094620    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.094706 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.094638    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196103    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196137    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196167    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196521 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196500    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196585 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196505    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196675 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196657    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hkf\" (UniqueName: \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-kube-api-access-h9hkf\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196769 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196697    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196769 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196738    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196861 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196779    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196861 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196815    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.196959 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.196903    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.197140 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.197116    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.197250 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.197228    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.197402 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.197380    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.199044 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.199016    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.199350 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.199331    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.219089 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.219069    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.219230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.219209    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hkf\" (UniqueName: \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-kube-api-access-h9hkf\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.242563 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.242538    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"]
Oct 30 02:56:44.245481 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.245461    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:44.247726 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.247710    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.279712 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.279584    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"]
Oct 30 02:56:44.399992 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.399954    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.400168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.400031    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.400168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.400124    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6n6\" (UniqueName: \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-kube-api-access-lx6n6\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.400294 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.400177    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.400294 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.400261    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.400395 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.400330    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.400395 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.400369    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.400501 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.400476    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.400550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.400537    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.501820 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.501745    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.501820 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.501790    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.501994 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.501830    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.501994 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.501860    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6n6\" (UniqueName: \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-kube-api-access-lx6n6\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.501994 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.501888    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.501994 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.501906    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.501994 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.501964    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.502198 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.501996    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.502198 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.502057    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.502362 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.502328    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.502593 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.502573    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.502686 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.502663    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.502746 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.502679    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.502746 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.502700    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.504263 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.504236    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.504419 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.504405    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.521822 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.521803    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.521966 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.521941    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6n6\" (UniqueName: \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-kube-api-access-lx6n6\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.575323 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.575301    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"]
Oct 30 02:56:44.576000 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:56:44.575977    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb58a83_97bd_4072_9fb4_8a9754fa3fdc.slice/crio-e62e724865765678b94f0e2e61ad1b3bba36185c5b684fb7876076c472e90d69 WatchSource:0}: Error finding container e62e724865765678b94f0e2e61ad1b3bba36185c5b684fb7876076c472e90d69: Status 404 returned error can't find the container with id e62e724865765678b94f0e2e61ad1b3bba36185c5b684fb7876076c472e90d69
Oct 30 02:56:44.578626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.578593    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:44.578701 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.578658    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:44.578701 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.578687    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:44.587649 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.587624    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:44.762820 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.762148    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" event={"ID":"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc","Type":"ContainerStarted","Data":"07e9289bf03a1c897ae879712a476f1122e809826c7dc24fe212ba67ffc1b2e2"}
Oct 30 02:56:44.762820 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.762199    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" event={"ID":"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc","Type":"ContainerStarted","Data":"e62e724865765678b94f0e2e61ad1b3bba36185c5b684fb7876076c472e90d69"}
Oct 30 02:56:44.811896 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.811844    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" podStartSLOduration=1.8118300280000001 podStartE2EDuration="1.811830028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:44.809656502 +0000 UTC m=+942.733864715" watchObservedRunningTime="2026-10-30 02:56:44.811830028 +0000 UTC m=+942.736038299"
Oct 30 02:56:44.825492 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.825461    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"]
Oct 30 02:56:44.829444 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.829409    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:44.855397 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.855365    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"]
Oct 30 02:56:44.908842 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.908808    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:44.908995 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.908874    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:44.908995 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.908890    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:44.909083 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.909002    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:44.909083 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.909052    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:44.909216 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.909199    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:44.909269 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.909254    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:44.909306 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.909286    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:44.909476 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.909453    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jjv4\" (UniqueName: \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-kube-api-access-5jjv4\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:44.914522 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.914498    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"]
Oct 30 02:56:44.915325 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:56:44.915304    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e529a9_14f9_4f80_96a2_2c56fd4abe64.slice/crio-847165ddd93acb98319ee9f0275fa61aa2ac3ef04f3aabf489f02e5122e05781 WatchSource:0}: Error finding container 847165ddd93acb98319ee9f0275fa61aa2ac3ef04f3aabf489f02e5122e05781: Status 404 returned error can't find the container with id 847165ddd93acb98319ee9f0275fa61aa2ac3ef04f3aabf489f02e5122e05781
Oct 30 02:56:44.918379 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.918351    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:44.918477 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.918409    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:44.918477 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:44.918453    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:45.011049 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011018    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.011524 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011107    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.011524 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011136    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.011524 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011168    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.011524 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011218    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.011524 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011269    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.011524 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011336    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.011524 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011371    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.011524 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011456    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jjv4\" (UniqueName: \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-kube-api-access-5jjv4\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.011901 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011649    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.011901 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.011698    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.012778 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.012756    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.013130 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.013020    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.013255 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.013232    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.013981 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.013957    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.014071 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.013965    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.031483 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.031451    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.032023 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.031999    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jjv4\" (UniqueName: \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-kube-api-access-5jjv4\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.143883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.143845    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:45.246497 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.246455    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:45.247568 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.247538    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:56:45.247679 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.247599    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 02:56:45.487184 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.487087    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"]
Oct 30 02:56:45.491137 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:56:45.491110    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3e5048_ada2_4a8e_bcff_66e9c831fd00.slice/crio-b84e4f56d02af4f169b7fb5065b63a982ef744df7e61c23273493cbc6b2399d2 WatchSource:0}: Error finding container b84e4f56d02af4f169b7fb5065b63a982ef744df7e61c23273493cbc6b2399d2: Status 404 returned error can't find the container with id b84e4f56d02af4f169b7fb5065b63a982ef744df7e61c23273493cbc6b2399d2
Oct 30 02:56:45.494183 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.494151    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:45.494270 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.494229    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:45.494334 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.494270    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:45.796450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.796133    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" event={"ID":"ff3e5048-ada2-4a8e-bcff-66e9c831fd00","Type":"ContainerStarted","Data":"2dfe3fac9ff05338e0ebfcd7c22a55c7c6c6ecbcbc38b599122fb13095dfa069"}
Oct 30 02:56:45.796450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.796203    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" event={"ID":"ff3e5048-ada2-4a8e-bcff-66e9c831fd00","Type":"ContainerStarted","Data":"b84e4f56d02af4f169b7fb5065b63a982ef744df7e61c23273493cbc6b2399d2"}
Oct 30 02:56:45.815685 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.812744    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" event={"ID":"04e529a9-14f9-4f80-96a2-2c56fd4abe64","Type":"ContainerStarted","Data":"7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f"}
Oct 30 02:56:45.815978 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.815931    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" event={"ID":"04e529a9-14f9-4f80-96a2-2c56fd4abe64","Type":"ContainerStarted","Data":"847165ddd93acb98319ee9f0275fa61aa2ac3ef04f3aabf489f02e5122e05781"}
Oct 30 02:56:45.849393 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.849332    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podStartSLOduration=1.849314946 podStartE2EDuration="1.849314946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:45.84851022 +0000 UTC m=+943.772718408" watchObservedRunningTime="2026-10-30 02:56:45.849314946 +0000 UTC m=+943.773523134"
Oct 30 02:56:45.908231 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:45.908165    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" podStartSLOduration=1.908145845 podStartE2EDuration="1.908145845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:45.904933743 +0000 UTC m=+943.829141933" watchObservedRunningTime="2026-10-30 02:56:45.908145845 +0000 UTC m=+943.832354032"
Oct 30 02:56:46.145249 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:46.145158    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:46.146621 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:46.146592    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 02:56:46.146770 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:46.146643    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 02:56:46.245982 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:46.245933    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:56:46.246179 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:46.245998    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 02:56:46.595485 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:46.592417    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:46.596550 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:46.596521    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:56:46.596763 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:46.596737    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 02:56:47.144945 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:47.144908    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 02:56:47.145134 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:47.144968    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 02:56:47.246823 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:47.246786    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:56:47.247211 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:47.246840    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 02:56:47.596449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:47.595865    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:56:47.596449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:47.595932    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 02:56:48.155447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:48.154948    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 02:56:48.155447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:48.155009    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 02:56:48.248918 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:48.248874    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:56:48.249491 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:48.249458    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 02:56:48.606458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:48.606355    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:56:48.606458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:48.606414    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 02:56:49.157364 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:49.152798    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 02:56:49.157364 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:49.152865    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 02:56:49.309450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:49.283772    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:56:49.309450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:49.283836    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 02:56:49.593452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:49.592544    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:56:49.593452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:49.592607    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 02:56:50.188664 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:50.188610    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 02:56:50.188982 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:50.188940    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 02:56:50.264749 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:50.264599    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:56:50.264749 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:50.264680    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 02:56:50.469712 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:50.469627    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:50.602874 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:50.602834    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:56:50.603148 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:50.603117    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 02:56:50.733455 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:50.733351    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:50.936542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:50.936509    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:56:51.145057 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.145026    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 02:56:51.145230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.145094    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 02:56:51.194775 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.194737    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"]
Oct 30 02:56:51.200146 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.200118    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.218561 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.218509    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"]
Oct 30 02:56:51.219087 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.219063    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"distinct-2-w3-d9f3ac-kserve-self-signed-certs\""
Oct 30 02:56:51.246717 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.246682    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:56:51.246893 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.246744    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 02:56:51.297189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.297152    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tls-certs\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.297373 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.297232    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-model-cache\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.297373 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.297261    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-dshm\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.297373 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.297345    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9xg\" (UniqueName: \"kubernetes.io/projected/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-kube-api-access-cp9xg\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.297585 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.297408    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tmp-dir\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.297585 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.297539    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-home\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.402596 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.402514    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-home\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.402752 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.402726    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tls-certs\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.402932 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.402907    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-model-cache\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.403160 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.403082    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-dshm\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.403489 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.403438    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9xg\" (UniqueName: \"kubernetes.io/projected/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-kube-api-access-cp9xg\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.403489 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.402973    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-home\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.403640 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.403595    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tmp-dir\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.403640 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.403229    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-model-cache\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.404070 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.404046    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tmp-dir\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.419162 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.419133    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tls-certs\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.421574 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.421526    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-dshm\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.463450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.456287    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9xg\" (UniqueName: \"kubernetes.io/projected/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-kube-api-access-cp9xg\") pod \"distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.536448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.535662    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:51.590451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.588809    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:56:51.590451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.588864    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 02:56:51.920791 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.920752    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"]
Oct 30 02:56:51.935927 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:51.935889    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" event={"ID":"764d1f39-ea66-4aa4-99ce-59cd2cda80b5","Type":"ContainerStarted","Data":"f5310163b4ef9faeb4ccdbdc53417a1a3e163304fe3ab7df80c54cb1b4033e31"}
Oct 30 02:56:52.144519 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:52.144300    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 02:56:52.144519 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:52.144355    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 02:56:52.289932 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:52.289873    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:52.291347 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:52.291321    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:52.292079 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:52.292058    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 02:56:52.645563 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:52.645482    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:52.646571 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:52.646551    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:52.647486 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:52.647250    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 02:56:52.945046 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:52.944943    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" event={"ID":"764d1f39-ea66-4aa4-99ce-59cd2cda80b5","Type":"ContainerStarted","Data":"da388f186cfbe9123891acc9cc64c6b56b5179d7b5d9d65926b0f035f93ac532"}
Oct 30 02:56:53.149553 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:53.149526    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:53.149865 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:53.149844    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:53.150574 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:53.150559    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 02:56:53.196207 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:53.196117    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podStartSLOduration=2.196102731 podStartE2EDuration="2.196102731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:52.993971235 +0000 UTC m=+950.918179424" watchObservedRunningTime="2026-10-30 02:56:53.196102731 +0000 UTC m=+951.120310921"
Oct 30 02:56:53.536220 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:53.536181    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:56:54.188277 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:56:54.188242    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:56:59.114570 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.114535    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"]
Oct 30 02:56:59.118744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.118726    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.154506 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.154477    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"]
Oct 30 02:56:59.228041 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.228011    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.228190 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.228045    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.228190 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.228156    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.228263 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.228217    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.228307 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.228286    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abb2acf-b64c-48d4-908a-2394f98f974f-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.228342 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.228317    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.228404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.228391    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mznm2\" (UniqueName: \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-kube-api-access-mznm2\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.228472 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.228460    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.228507 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.228497    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.272352 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.272323    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"]
Oct 30 02:56:59.276285 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.276268    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.299344 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.299320    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"]
Oct 30 02:56:59.329270 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329239    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.329391 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329288    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abb2acf-b64c-48d4-908a-2394f98f974f-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.329391 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329319    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.329391 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329348    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.329585 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329395    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.329585 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329539    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.329708 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329638    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mznm2\" (UniqueName: \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-kube-api-access-mznm2\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.329708 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329672    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.329810 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329743    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.329810 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329751    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.329810 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329778    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf6th\" (UniqueName: \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-kube-api-access-mf6th\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.329931 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329823    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.330015 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.329992    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.330069 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.330053    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.330137 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.330111    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.330192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.330135    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abb2acf-b64c-48d4-908a-2394f98f974f-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.330192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.330159    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.330690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.330666    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.330820 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.330698    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.330903 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.330789    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.330979 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.330959    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.331034 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.330991    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.331315 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.331293    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.332735 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.332710    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.332856 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.332713    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.350231 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.350206    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mznm2\" (UniqueName: \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-kube-api-access-mznm2\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.350980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.350959    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.432380 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432318    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.432380 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432366    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.432583 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432416    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.432583 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432459    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.432583 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432485    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.432583 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432516    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.432583 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432563    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf6th\" (UniqueName: \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-kube-api-access-mf6th\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.432844 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432641    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.432844 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432673    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.432844 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432733    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.432999 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.432917    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.433142 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.433118    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.433203 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.433123    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.433203 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.433148    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.434295 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.434273    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:56:59.434662 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.434642    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.434874 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.434858    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.453622 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.453602    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.453869 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.453848    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf6th\" (UniqueName: \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-kube-api-access-mf6th\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.589142 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.589109    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:56:59.753436 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.753389    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"]
Oct 30 02:56:59.754140 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:56:59.754113    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abb2acf_b64c_48d4_908a_2394f98f974f.slice/crio-fe6bd1ee7c69f1a815ba354f705315661d60bf3b538a57ae6edb0bae09c2a16f WatchSource:0}: Error finding container fe6bd1ee7c69f1a815ba354f705315661d60bf3b538a57ae6edb0bae09c2a16f: Status 404 returned error can't find the container with id fe6bd1ee7c69f1a815ba354f705315661d60bf3b538a57ae6edb0bae09c2a16f
Oct 30 02:56:59.756460 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.756398    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:59.756534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.756489    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:59.756534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.756524    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:59.892368 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.892342    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"]
Oct 30 02:56:59.894695 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:56:59.894665    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2df6b23_0d39_4222_b9b1_31568f4df8a9.slice/crio-1397603c8454f7f61f1a556e4f890ea27c3d8df63db8f8f071c91176ee7d0b36 WatchSource:0}: Error finding container 1397603c8454f7f61f1a556e4f890ea27c3d8df63db8f8f071c91176ee7d0b36: Status 404 returned error can't find the container with id 1397603c8454f7f61f1a556e4f890ea27c3d8df63db8f8f071c91176ee7d0b36
Oct 30 02:56:59.897336 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.897300    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:59.897453 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.897369    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:59.897453 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:56:59.897399    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:00.003151 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.003050    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" event={"ID":"6abb2acf-b64c-48d4-908a-2394f98f974f","Type":"ContainerStarted","Data":"c9adb7167f53fac1f0708688857383d148e727a1b671c6360ffedfd92e34e7a9"}
Oct 30 02:57:00.003151 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.003116    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" event={"ID":"6abb2acf-b64c-48d4-908a-2394f98f974f","Type":"ContainerStarted","Data":"fe6bd1ee7c69f1a815ba354f705315661d60bf3b538a57ae6edb0bae09c2a16f"}
Oct 30 02:57:00.005047 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.005021    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" event={"ID":"b2df6b23-0d39-4222-b9b1-31568f4df8a9","Type":"ContainerStarted","Data":"71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d"}
Oct 30 02:57:00.005185 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.005065    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" event={"ID":"b2df6b23-0d39-4222-b9b1-31568f4df8a9","Type":"ContainerStarted","Data":"1397603c8454f7f61f1a556e4f890ea27c3d8df63db8f8f071c91176ee7d0b36"}
Oct 30 02:57:00.056532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.056461    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" podStartSLOduration=1.056441265 podStartE2EDuration="1.056441265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:00.052901119 +0000 UTC m=+957.977109316" watchObservedRunningTime="2026-10-30 02:57:00.056441265 +0000 UTC m=+957.980649453"
Oct 30 02:57:00.097541 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.096278    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" podStartSLOduration=1.096259666 podStartE2EDuration="1.096259666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:00.095957385 +0000 UTC m=+958.020165606" watchObservedRunningTime="2026-10-30 02:57:00.096259666 +0000 UTC m=+958.020467854"
Oct 30 02:57:00.438687 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.438653    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:57:00.440132 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.440096    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:57:00.440288 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.440146    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 02:57:00.592446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.591752    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:57:00.599447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.597668    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:57:00.599447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:00.597727    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 02:57:01.434907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:01.434758    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:57:01.434907 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:01.434825    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 02:57:01.536200 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:01.536167    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:57:01.548630 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:01.548607    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:57:01.593053 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:01.593007    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:57:01.593245 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:01.593076    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 02:57:02.037904 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:02.037872    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:57:02.434816 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:02.434734    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:57:02.434816 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:02.434788    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 02:57:02.600206 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:02.600165    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:57:02.600784 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:02.600752    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 02:57:03.434966 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:03.434927    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:57:03.435134 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:03.434976    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 02:57:03.591447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:03.590852    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:57:03.591447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:03.590917    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 02:57:04.435248 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:04.435214    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:57:04.435674 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:04.435266    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 02:57:04.596626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:04.596589    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:57:04.596997 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:04.596964    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 02:57:05.435287 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:05.435245    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:57:05.435657 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:05.435307    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 02:57:05.594390 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:05.593878    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:57:05.594390 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:05.593941    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 02:57:06.434987 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:06.434944    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:57:06.435134 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:06.435034    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 02:57:06.446325 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:06.446294    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:57:06.589691 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:06.589650    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:57:06.589850 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:06.589729    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 02:57:07.440660 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:07.440598    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:57:07.441061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:07.441033    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:57:07.442786 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:07.442765    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 02:57:07.628448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:07.605701    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:57:07.649839 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:07.649812    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:57:07.663710 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:07.663678    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 02:57:08.953562 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:08.953516    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:08.979579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:08.979519    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.029579 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:09.029537    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.050538 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:09.050496    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.683448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:09.682557    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.711448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:09.711031    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.055449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:10.051530    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.055449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:10.052399    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:57:10.146615 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:10.146352    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:57:24.111848 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.111808    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"]
Oct 30 02:57:24.116105 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.116080    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.134163 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.134137    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"]
Oct 30 02:57:24.142889 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.142870    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:57:24.143499 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.143483    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:57:24.189134 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:57:24.189099    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:57:24.214273 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.214245    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82edc560-hpvx8\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.214410 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.214297    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82edc560-hpvx8\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.214410 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.214392    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxh4r\" (UniqueName: \"kubernetes.io/projected/63c22982-e253-4d6f-ab95-1f0f15af4db0-kube-api-access-zxh4r\") pod \"maas-api-revoke-sub-82edc560-hpvx8\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.315448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.315401    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82edc560-hpvx8\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.315595 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.315501    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxh4r\" (UniqueName: \"kubernetes.io/projected/63c22982-e253-4d6f-ab95-1f0f15af4db0-kube-api-access-zxh4r\") pod \"maas-api-revoke-sub-82edc560-hpvx8\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.315740 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.315721    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82edc560-hpvx8\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.315976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.315956    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82edc560-hpvx8\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.317859 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.317837    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82edc560-hpvx8\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.336726 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.336701    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxh4r\" (UniqueName: \"kubernetes.io/projected/63c22982-e253-4d6f-ab95-1f0f15af4db0-kube-api-access-zxh4r\") pod \"maas-api-revoke-sub-82edc560-hpvx8\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.429616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.429551    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:24.726114 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:24.726090    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"]
Oct 30 02:57:24.727548 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:57:24.727522    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c22982_e253_4d6f_ab95_1f0f15af4db0.slice/crio-77ecd1a25868835ddf2ad79860a712619153468f25f98143b81b95de418b5acd WatchSource:0}: Error finding container 77ecd1a25868835ddf2ad79860a712619153468f25f98143b81b95de418b5acd: Status 404 returned error can't find the container with id 77ecd1a25868835ddf2ad79860a712619153468f25f98143b81b95de418b5acd
Oct 30 02:57:25.244187 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:25.244102    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8" event={"ID":"63c22982-e253-4d6f-ab95-1f0f15af4db0","Type":"ContainerStarted","Data":"1b4e5a7b5b022f469944bf96943a898facefccff6c11d72c8556ce48736dda68"}
Oct 30 02:57:25.244187 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:25.244154    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8" event={"ID":"63c22982-e253-4d6f-ab95-1f0f15af4db0","Type":"ContainerStarted","Data":"77ecd1a25868835ddf2ad79860a712619153468f25f98143b81b95de418b5acd"}
Oct 30 02:57:25.303639 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:25.303590    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8" podStartSLOduration=1.303578881 podStartE2EDuration="1.303578881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:25.28895945 +0000 UTC m=+983.213167635" watchObservedRunningTime="2026-10-30 02:57:25.303578881 +0000 UTC m=+983.227787066"
Oct 30 02:57:26.250836 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:26.250804    4278 generic.go:356] "Generic (PLEG): container finished" podID="63c22982-e253-4d6f-ab95-1f0f15af4db0" containerID="1b4e5a7b5b022f469944bf96943a898facefccff6c11d72c8556ce48736dda68" exitCode=0
Oct 30 02:57:26.251229 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:26.250880    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8" event={"ID":"63c22982-e253-4d6f-ab95-1f0f15af4db0","Type":"ContainerDied","Data":"1b4e5a7b5b022f469944bf96943a898facefccff6c11d72c8556ce48736dda68"}
Oct 30 02:57:27.417859 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:27.417831    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:27.555877 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:27.555791    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-openshift-ca\") pod \"63c22982-e253-4d6f-ab95-1f0f15af4db0\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") "
Oct 30 02:57:27.555877 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:27.555867    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-xks-ca\") pod \"63c22982-e253-4d6f-ab95-1f0f15af4db0\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") "
Oct 30 02:57:27.556116 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:27.556015    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c22982-e253-4d6f-ab95-1f0f15af4db0-kube-api-access-zxh4r\" (UniqueName: \"kubernetes.io/projected/63c22982-e253-4d6f-ab95-1f0f15af4db0-kube-api-access-zxh4r\") pod \"63c22982-e253-4d6f-ab95-1f0f15af4db0\" (UID: \"63c22982-e253-4d6f-ab95-1f0f15af4db0\") "
Oct 30 02:57:27.556217 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:27.556194    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-openshift-ca" pod "63c22982-e253-4d6f-ab95-1f0f15af4db0" (UID: "63c22982-e253-4d6f-ab95-1f0f15af4db0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:27.556540 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:27.556521    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:27.557883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:27.557859    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-xks-ca" pod "63c22982-e253-4d6f-ab95-1f0f15af4db0" (UID: "63c22982-e253-4d6f-ab95-1f0f15af4db0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:27.557981 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:27.557912    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c22982-e253-4d6f-ab95-1f0f15af4db0-kube-api-access-zxh4r" pod "63c22982-e253-4d6f-ab95-1f0f15af4db0" (UID: "63c22982-e253-4d6f-ab95-1f0f15af4db0"). InnerVolumeSpecName "kube-api-access-zxh4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:27.657771 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:27.657743    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c22982-e253-4d6f-ab95-1f0f15af4db0-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:27.657918 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:27.657787    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxh4r\" (UniqueName: \"kubernetes.io/projected/63c22982-e253-4d6f-ab95-1f0f15af4db0-kube-api-access-zxh4r\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:28.263858 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:28.263825    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"
Oct 30 02:57:28.263858 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:28.263832    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8" event={"ID":"63c22982-e253-4d6f-ab95-1f0f15af4db0","Type":"ContainerDied","Data":"77ecd1a25868835ddf2ad79860a712619153468f25f98143b81b95de418b5acd"}
Oct 30 02:57:28.264072 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:28.263887    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77ecd1a25868835ddf2ad79860a712619153468f25f98143b81b95de418b5acd"
Oct 30 02:57:29.017230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.017195    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"]
Oct 30 02:57:29.017944 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.017921    4278 cpu_manager.go:414] "containerMap: removing container" podUID="63c22982-e253-4d6f-ab95-1f0f15af4db0" containerName="revoke-keys"
Oct 30 02:57:29.018032 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.017945    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c22982-e253-4d6f-ab95-1f0f15af4db0" containerName="revoke-keys"
Oct 30 02:57:29.018114 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.018101    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c22982-e253-4d6f-ab95-1f0f15af4db0" containerName="revoke-keys"
Oct 30 02:57:29.021877 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.021858    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.042660 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.042637    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"]
Oct 30 02:57:29.174609 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.174574    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.174781 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.174691    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.174781 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.174734    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.174897 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.174850    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096fb20-7117-439d-8901-7e9e9321c044-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.174993 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.174977    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096fb20-7117-439d-8901-7e9e9321c044-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.175053 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.175009    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.175101 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.175087    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.175140 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.175125    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jnn\" (UniqueName: \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-kube-api-access-g7jnn\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.175194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.175152    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.199707 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.199682    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"]
Oct 30 02:57:29.203946 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.203923    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.228555 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.228528    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"]
Oct 30 02:57:29.276228 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276151    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276228 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276187    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276463 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276267    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096fb20-7117-439d-8901-7e9e9321c044-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276463 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276369    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096fb20-7117-439d-8901-7e9e9321c044-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276463 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276399    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276632 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276491    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276632 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276542    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jnn\" (UniqueName: \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-kube-api-access-g7jnn\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276632 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276549    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276632 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276575    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276816 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276636    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276860 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276812    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.276986 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.276955    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.277110 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.277065    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096fb20-7117-439d-8901-7e9e9321c044-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.277174 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.277112    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.278734 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.278713    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.278865 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.278846    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096fb20-7117-439d-8901-7e9e9321c044-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.296706 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.296674    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.296792 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.296743    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jnn\" (UniqueName: \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-kube-api-access-g7jnn\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.336121 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.336098    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:29.378847 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.378811    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.379008 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.378981    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.379076 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.379055    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.379161 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.379123    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.379161 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.379149    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.379276 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.379200    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.379641 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.379606    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6gg7\" (UniqueName: \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-kube-api-access-w6gg7\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.379733 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.379667    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443f59af-9b47-43ea-91b1-63054fcd0ec7-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.379823 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.379800    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.481107 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481069    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.481364 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481222    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.481364 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481283    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.481364 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481331    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.481364 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481353    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.481752 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481400    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.481752 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481543    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6gg7\" (UniqueName: \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-kube-api-access-w6gg7\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.481752 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481575    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443f59af-9b47-43ea-91b1-63054fcd0ec7-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.481752 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481622    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.482003 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481964    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.482003 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.481986    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.482282 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.482235    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.482541 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.482510    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443f59af-9b47-43ea-91b1-63054fcd0ec7-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.482770 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.482749    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.484770 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.484753    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.484905 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.484884    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.501438 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.501405    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.501546 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.501497    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6gg7\" (UniqueName: \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-kube-api-access-w6gg7\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.521474 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.521442    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:29.653997 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.653967    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"]
Oct 30 02:57:29.658748 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.658718    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.658827 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.658782    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.658827 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.658817    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.828691 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.828668    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"]
Oct 30 02:57:29.831166 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:57:29.831129    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443f59af_9b47_43ea_91b1_63054fcd0ec7.slice/crio-ba4f2775eafed438ef6a4844c46fb7a4524817a7d2659136d5c16b36ef95a873 WatchSource:0}: Error finding container ba4f2775eafed438ef6a4844c46fb7a4524817a7d2659136d5c16b36ef95a873: Status 404 returned error can't find the container with id ba4f2775eafed438ef6a4844c46fb7a4524817a7d2659136d5c16b36ef95a873
Oct 30 02:57:29.833805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.833769    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.833873 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.833849    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.833915 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:29.833889    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:30.314314 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.314278    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" event={"ID":"b096fb20-7117-439d-8901-7e9e9321c044","Type":"ContainerStarted","Data":"9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511"}
Oct 30 02:57:30.314738 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.314374    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" event={"ID":"b096fb20-7117-439d-8901-7e9e9321c044","Type":"ContainerStarted","Data":"03122a68df48003a2b7f52272d3b4cb573089d575f63364e13c2c1048979bbe9"}
Oct 30 02:57:30.316202 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.315849    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" event={"ID":"443f59af-9b47-43ea-91b1-63054fcd0ec7","Type":"ContainerStarted","Data":"6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712"}
Oct 30 02:57:30.316202 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.315888    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" event={"ID":"443f59af-9b47-43ea-91b1-63054fcd0ec7","Type":"ContainerStarted","Data":"ba4f2775eafed438ef6a4844c46fb7a4524817a7d2659136d5c16b36ef95a873"}
Oct 30 02:57:30.340988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.336583    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:30.340988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.338143    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:57:30.340988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.338198    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 02:57:30.384830 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.384771    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podStartSLOduration=2.3847498209999998 podStartE2EDuration="2.384749821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:30.358341246 +0000 UTC m=+988.282549434" watchObservedRunningTime="2026-10-30 02:57:30.384749821 +0000 UTC m=+988.308958010"
Oct 30 02:57:30.419013 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.418950    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podStartSLOduration=1.418917828 podStartE2EDuration="1.418917828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:30.413649025 +0000 UTC m=+988.337857213" watchObservedRunningTime="2026-10-30 02:57:30.418917828 +0000 UTC m=+988.343126016"
Oct 30 02:57:30.521760 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.521722    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:30.522931 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.522907    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:57:30.523028 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:30.522958    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 02:57:31.336980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:31.336937    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:57:31.337375 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:31.336996    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 02:57:31.521933 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:31.521897    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:57:31.522110 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:31.521948    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 02:57:32.337022 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:32.336979    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:57:32.337481 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:32.337037    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 02:57:32.522700 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:32.522668    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:57:32.522865 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:32.522717    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 02:57:33.336573 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:33.336539    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:57:33.336759 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:33.336590    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 02:57:33.522858 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:33.522821    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:57:33.523260 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:33.522875    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 02:57:34.336666 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:34.336629    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:57:34.336861 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:34.336703    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 02:57:34.522095 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:34.522058    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:57:34.522260 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:34.522120    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 02:57:35.336615 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:35.336581    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:57:35.337024 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:35.336630    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 02:57:35.522064 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:35.522027    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:57:35.522241 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:35.522092    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 02:57:36.336372 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.336338    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:57:36.336594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.336387    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 02:57:36.522515 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.522473    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:57:36.522931 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.522548    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 02:57:36.641076 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.640971    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"]
Oct 30 02:57:36.647931 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.647903    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:36.666519 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.666234    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"]
Oct 30 02:57:36.674122 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.674099    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:57:36.674267 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.674202    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:57:36.734641 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.732260    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9e64a7-68cd9\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:36.735045 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.734998    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22zs2\" (UniqueName: \"kubernetes.io/projected/2374aaf2-f128-4912-9894-8e71ff9b7c5d-kube-api-access-22zs2\") pod \"maas-api-revoke-sub-6e9e64a7-68cd9\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:36.735315 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.735244    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9e64a7-68cd9\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:36.836874 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.836836    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22zs2\" (UniqueName: \"kubernetes.io/projected/2374aaf2-f128-4912-9894-8e71ff9b7c5d-kube-api-access-22zs2\") pod \"maas-api-revoke-sub-6e9e64a7-68cd9\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:36.837035 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.836924    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9e64a7-68cd9\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:36.837121 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.837068    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9e64a7-68cd9\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:36.837962 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.837931    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9e64a7-68cd9\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:36.840053 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.840023    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9e64a7-68cd9\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:36.858845 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.858811    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22zs2\" (UniqueName: \"kubernetes.io/projected/2374aaf2-f128-4912-9894-8e71ff9b7c5d-kube-api-access-22zs2\") pod \"maas-api-revoke-sub-6e9e64a7-68cd9\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:36.976988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:36.976894    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:37.288263 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:37.288230    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"]
Oct 30 02:57:37.288739 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:57:37.288716    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2374aaf2_f128_4912_9894_8e71ff9b7c5d.slice/crio-dd16deb9603fbfd1531bafdaaf2eff1903e20da30c968dd374902cfaee3b7f32 WatchSource:0}: Error finding container dd16deb9603fbfd1531bafdaaf2eff1903e20da30c968dd374902cfaee3b7f32: Status 404 returned error can't find the container with id dd16deb9603fbfd1531bafdaaf2eff1903e20da30c968dd374902cfaee3b7f32
Oct 30 02:57:37.341180 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:37.341160    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:37.341632 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:37.341614    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:37.342375 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:37.342359    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 02:57:37.366305 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:37.366282    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9" event={"ID":"2374aaf2-f128-4912-9894-8e71ff9b7c5d","Type":"ContainerStarted","Data":"dd16deb9603fbfd1531bafdaaf2eff1903e20da30c968dd374902cfaee3b7f32"}
Oct 30 02:57:37.526043 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:37.526017    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:37.526490 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:37.526457    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:37.527219 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:37.527200    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 02:57:38.373435 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:38.373392    4278 generic.go:356] "Generic (PLEG): container finished" podID="2374aaf2-f128-4912-9894-8e71ff9b7c5d" containerID="ef021672b5b48c6cb93d43d935a3c353aac13ce042b9334f9228c21e36a3d5e3" exitCode=0
Oct 30 02:57:38.373585 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:38.373457    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9" event={"ID":"2374aaf2-f128-4912-9894-8e71ff9b7c5d","Type":"ContainerDied","Data":"ef021672b5b48c6cb93d43d935a3c353aac13ce042b9334f9228c21e36a3d5e3"}
Oct 30 02:57:39.263216 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.263179    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:57:39.525771 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.525714    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:39.575528 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.575502    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-openshift-ca\") pod \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") "
Oct 30 02:57:39.575675 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.575632    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2374aaf2-f128-4912-9894-8e71ff9b7c5d-kube-api-access-22zs2\" (UniqueName: \"kubernetes.io/projected/2374aaf2-f128-4912-9894-8e71ff9b7c5d-kube-api-access-22zs2\") pod \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") "
Oct 30 02:57:39.575736 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.575714    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-xks-ca\") pod \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\" (UID: \"2374aaf2-f128-4912-9894-8e71ff9b7c5d\") "
Oct 30 02:57:39.575909 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.575885    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-openshift-ca" pod "2374aaf2-f128-4912-9894-8e71ff9b7c5d" (UID: "2374aaf2-f128-4912-9894-8e71ff9b7c5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:39.576067 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.576048    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:39.577655 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.577625    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2374aaf2-f128-4912-9894-8e71ff9b7c5d-kube-api-access-22zs2" pod "2374aaf2-f128-4912-9894-8e71ff9b7c5d" (UID: "2374aaf2-f128-4912-9894-8e71ff9b7c5d"). InnerVolumeSpecName "kube-api-access-22zs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:39.577775 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.577677    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-xks-ca" pod "2374aaf2-f128-4912-9894-8e71ff9b7c5d" (UID: "2374aaf2-f128-4912-9894-8e71ff9b7c5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:39.676874 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.676831    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22zs2\" (UniqueName: \"kubernetes.io/projected/2374aaf2-f128-4912-9894-8e71ff9b7c5d-kube-api-access-22zs2\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:39.676874 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:39.676858    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2374aaf2-f128-4912-9894-8e71ff9b7c5d-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:40.390145 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:40.390104    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9" event={"ID":"2374aaf2-f128-4912-9894-8e71ff9b7c5d","Type":"ContainerDied","Data":"dd16deb9603fbfd1531bafdaaf2eff1903e20da30c968dd374902cfaee3b7f32"}
Oct 30 02:57:40.390687 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:40.390170    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd16deb9603fbfd1531bafdaaf2eff1903e20da30c968dd374902cfaee3b7f32"
Oct 30 02:57:40.390687 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:40.390117    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"
Oct 30 02:57:42.193449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:42.192553    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.459507 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:42.458538    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.482450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:42.481562    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.735530 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:42.729293    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.764669 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:42.763960    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.933489 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:42.932534    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.961095 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:42.960522    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 02:57:42.994626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:42.994530    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:43.794447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:43.793555    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:43.794447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:43.794339    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:57:43.847448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:43.840538    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 02:57:43.847448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:43.842260    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:57:44.001458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:43.988227    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.001458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:43.988635    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:57:44.001458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:43.989153    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.010459 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:44.002488    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:44.033875 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:44.033052    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:44.044518 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:43.989493    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:57:44.044518 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:44.038556    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:57:44.209476 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:44.178851    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:44.430007 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:44.429933    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:57:44.432979 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:44.432859    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:57:44.489987 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:44.489953    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:57:44.580443 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:44.560538    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:57:44.580443 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:44.565852    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:57:47.610339 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.610300    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"]
Oct 30 02:57:47.611144 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.611122    4278 cpu_manager.go:414] "containerMap: removing container" podUID="2374aaf2-f128-4912-9894-8e71ff9b7c5d" containerName="revoke-keys"
Oct 30 02:57:47.611230 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.611145    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2374aaf2-f128-4912-9894-8e71ff9b7c5d" containerName="revoke-keys"
Oct 30 02:57:47.611326 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.611315    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="2374aaf2-f128-4912-9894-8e71ff9b7c5d" containerName="revoke-keys"
Oct 30 02:57:47.616094 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.616076    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:47.631874 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.631847    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"]
Oct 30 02:57:47.653243 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.653214    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:57:47.653489 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.653467    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:57:47.697909 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.697882    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbpzp\" (UniqueName: \"kubernetes.io/projected/a74730ba-4801-4ff7-ba08-1a080440a524-kube-api-access-pbpzp\") pod \"maas-api-revoke-sub-b0d749b6-g9l65\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:47.698063 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.698041    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d749b6-g9l65\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:47.698116 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.698102    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d749b6-g9l65\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:47.799793 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.799762    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d749b6-g9l65\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:47.799944 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.799825    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d749b6-g9l65\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:47.799988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.799946    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpzp\" (UniqueName: \"kubernetes.io/projected/a74730ba-4801-4ff7-ba08-1a080440a524-kube-api-access-pbpzp\") pod \"maas-api-revoke-sub-b0d749b6-g9l65\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:47.800592 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.800573    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d749b6-g9l65\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:47.802212 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.802192    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d749b6-g9l65\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:47.820328 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.820299    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpzp\" (UniqueName: \"kubernetes.io/projected/a74730ba-4801-4ff7-ba08-1a080440a524-kube-api-access-pbpzp\") pod \"maas-api-revoke-sub-b0d749b6-g9l65\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:47.932129 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:47.932049    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:48.238614 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:48.238585    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"]
Oct 30 02:57:48.239463 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:57:48.239419    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74730ba_4801_4ff7_ba08_1a080440a524.slice/crio-5d6a497b327905cb5888f613db1a6d8eed3c9d5ddf652198b1732680cec38aed WatchSource:0}: Error finding container 5d6a497b327905cb5888f613db1a6d8eed3c9d5ddf652198b1732680cec38aed: Status 404 returned error can't find the container with id 5d6a497b327905cb5888f613db1a6d8eed3c9d5ddf652198b1732680cec38aed
Oct 30 02:57:48.856137 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:48.856057    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65" event={"ID":"a74730ba-4801-4ff7-ba08-1a080440a524","Type":"ContainerStarted","Data":"5e5f67b8e17bfead4b407c4c4e2f4b3ddfa7a5dcc8ce30b9c53c4bb768f4848e"}
Oct 30 02:57:48.856137 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:48.856128    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65" event={"ID":"a74730ba-4801-4ff7-ba08-1a080440a524","Type":"ContainerStarted","Data":"5d6a497b327905cb5888f613db1a6d8eed3c9d5ddf652198b1732680cec38aed"}
Oct 30 02:57:48.900565 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:48.900509    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65" podStartSLOduration=1.900496941 podStartE2EDuration="1.900496941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:48.900095826 +0000 UTC m=+1006.824304014" watchObservedRunningTime="2026-10-30 02:57:48.900496941 +0000 UTC m=+1006.824705127"
Oct 30 02:57:49.863859 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:49.863827    4278 generic.go:356] "Generic (PLEG): container finished" podID="a74730ba-4801-4ff7-ba08-1a080440a524" containerID="5e5f67b8e17bfead4b407c4c4e2f4b3ddfa7a5dcc8ce30b9c53c4bb768f4848e" exitCode=0
Oct 30 02:57:49.864265 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:49.863905    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65" event={"ID":"a74730ba-4801-4ff7-ba08-1a080440a524","Type":"ContainerDied","Data":"5e5f67b8e17bfead4b407c4c4e2f4b3ddfa7a5dcc8ce30b9c53c4bb768f4848e"}
Oct 30 02:57:51.008397 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.008370    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:51.047515 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.047481    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-xks-ca\") pod \"a74730ba-4801-4ff7-ba08-1a080440a524\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") "
Oct 30 02:57:51.047736 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.047719    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-openshift-ca\") pod \"a74730ba-4801-4ff7-ba08-1a080440a524\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") "
Oct 30 02:57:51.047808 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.047794    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74730ba-4801-4ff7-ba08-1a080440a524-kube-api-access-pbpzp\" (UniqueName: \"kubernetes.io/projected/a74730ba-4801-4ff7-ba08-1a080440a524-kube-api-access-pbpzp\") pod \"a74730ba-4801-4ff7-ba08-1a080440a524\" (UID: \"a74730ba-4801-4ff7-ba08-1a080440a524\") "
Oct 30 02:57:51.048084 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.048055    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-openshift-ca" pod "a74730ba-4801-4ff7-ba08-1a080440a524" (UID: "a74730ba-4801-4ff7-ba08-1a080440a524"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:51.048325 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.048305    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:51.049771 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.049745    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-xks-ca" pod "a74730ba-4801-4ff7-ba08-1a080440a524" (UID: "a74730ba-4801-4ff7-ba08-1a080440a524"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:51.050064 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.050038    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74730ba-4801-4ff7-ba08-1a080440a524-kube-api-access-pbpzp" pod "a74730ba-4801-4ff7-ba08-1a080440a524" (UID: "a74730ba-4801-4ff7-ba08-1a080440a524"). InnerVolumeSpecName "kube-api-access-pbpzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:51.149718 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.149638    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74730ba-4801-4ff7-ba08-1a080440a524-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:51.149718 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.149667    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbpzp\" (UniqueName: \"kubernetes.io/projected/a74730ba-4801-4ff7-ba08-1a080440a524-kube-api-access-pbpzp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:51.876663 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.876629    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65" event={"ID":"a74730ba-4801-4ff7-ba08-1a080440a524","Type":"ContainerDied","Data":"5d6a497b327905cb5888f613db1a6d8eed3c9d5ddf652198b1732680cec38aed"}
Oct 30 02:57:51.876850 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.876688    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d6a497b327905cb5888f613db1a6d8eed3c9d5ddf652198b1732680cec38aed"
Oct 30 02:57:51.876850 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:51.876645    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"
Oct 30 02:57:53.035522 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.035481    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"]
Oct 30 02:57:53.036305 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.036281    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a74730ba-4801-4ff7-ba08-1a080440a524" containerName="revoke-keys"
Oct 30 02:57:53.036402 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.036305    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74730ba-4801-4ff7-ba08-1a080440a524" containerName="revoke-keys"
Oct 30 02:57:53.036470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.036447    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74730ba-4801-4ff7-ba08-1a080440a524" containerName="revoke-keys"
Oct 30 02:57:53.040059 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.040037    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.058469 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.058441    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"]
Oct 30 02:57:53.067181 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.067162    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:57:53.067259 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.067165    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:57:53.175396 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.175354    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5bkl\" (UniqueName: \"kubernetes.io/projected/675d2806-7d9f-4b20-8138-943eb40781e2-kube-api-access-r5bkl\") pod \"maas-api-revoke-sub-89a1b293-hmf8j\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.175590 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.175437    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a1b293-hmf8j\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.175590 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.175509    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a1b293-hmf8j\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.276922 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.276891    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5bkl\" (UniqueName: \"kubernetes.io/projected/675d2806-7d9f-4b20-8138-943eb40781e2-kube-api-access-r5bkl\") pod \"maas-api-revoke-sub-89a1b293-hmf8j\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.277078 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.276935    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a1b293-hmf8j\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.277078 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.276951    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a1b293-hmf8j\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.277592 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.277563    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a1b293-hmf8j\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.279213 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.279189    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a1b293-hmf8j\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.298304 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.298247    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5bkl\" (UniqueName: \"kubernetes.io/projected/675d2806-7d9f-4b20-8138-943eb40781e2-kube-api-access-r5bkl\") pod \"maas-api-revoke-sub-89a1b293-hmf8j\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.352584 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.352561    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:53.653999 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:57:53.653964    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675d2806_7d9f_4b20_8138_943eb40781e2.slice/crio-1b3abe5b3cbce37747d0e19dcba242223ae9443dc389f82085b7e9c3c368200a WatchSource:0}: Error finding container 1b3abe5b3cbce37747d0e19dcba242223ae9443dc389f82085b7e9c3c368200a: Status 404 returned error can't find the container with id 1b3abe5b3cbce37747d0e19dcba242223ae9443dc389f82085b7e9c3c368200a
Oct 30 02:57:53.654138 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.654103    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"]
Oct 30 02:57:53.889815 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:53.889780    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j" event={"ID":"675d2806-7d9f-4b20-8138-943eb40781e2","Type":"ContainerStarted","Data":"1b3abe5b3cbce37747d0e19dcba242223ae9443dc389f82085b7e9c3c368200a"}
Oct 30 02:57:54.188059 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:57:54.188018    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:57:54.897374 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:54.897344    4278 generic.go:356] "Generic (PLEG): container finished" podID="675d2806-7d9f-4b20-8138-943eb40781e2" containerID="45c3ca1b08ab5b37a50f9fa081dd9f59bd07345a7bfba56781761d5618f85d4b" exitCode=0
Oct 30 02:57:54.897548 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:54.897437    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j" event={"ID":"675d2806-7d9f-4b20-8138-943eb40781e2","Type":"ContainerDied","Data":"45c3ca1b08ab5b37a50f9fa081dd9f59bd07345a7bfba56781761d5618f85d4b"}
Oct 30 02:57:56.433455 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.424916    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:56.487404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.484651    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675d2806-7d9f-4b20-8138-943eb40781e2-kube-api-access-r5bkl\" (UniqueName: \"kubernetes.io/projected/675d2806-7d9f-4b20-8138-943eb40781e2-kube-api-access-r5bkl\") pod \"675d2806-7d9f-4b20-8138-943eb40781e2\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") "
Oct 30 02:57:56.487404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.484768    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-xks-ca\") pod \"675d2806-7d9f-4b20-8138-943eb40781e2\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") "
Oct 30 02:57:56.487404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.484868    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-openshift-ca\") pod \"675d2806-7d9f-4b20-8138-943eb40781e2\" (UID: \"675d2806-7d9f-4b20-8138-943eb40781e2\") "
Oct 30 02:57:56.487404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.486831    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-openshift-ca" pod "675d2806-7d9f-4b20-8138-943eb40781e2" (UID: "675d2806-7d9f-4b20-8138-943eb40781e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:56.624092 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.623771    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:56.672173 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.671888    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675d2806-7d9f-4b20-8138-943eb40781e2-kube-api-access-r5bkl" pod "675d2806-7d9f-4b20-8138-943eb40781e2" (UID: "675d2806-7d9f-4b20-8138-943eb40781e2"). InnerVolumeSpecName "kube-api-access-r5bkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:56.678578 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.678540    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-xks-ca" pod "675d2806-7d9f-4b20-8138-943eb40781e2" (UID: "675d2806-7d9f-4b20-8138-943eb40781e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:56.728700 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.728661    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5bkl\" (UniqueName: \"kubernetes.io/projected/675d2806-7d9f-4b20-8138-943eb40781e2-kube-api-access-r5bkl\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:56.728931 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.728913    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d2806-7d9f-4b20-8138-943eb40781e2-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:57:56.933508 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.929321    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j" event={"ID":"675d2806-7d9f-4b20-8138-943eb40781e2","Type":"ContainerDied","Data":"1b3abe5b3cbce37747d0e19dcba242223ae9443dc389f82085b7e9c3c368200a"}
Oct 30 02:57:56.933508 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.929417    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b3abe5b3cbce37747d0e19dcba242223ae9443dc389f82085b7e9c3c368200a"
Oct 30 02:57:56.933508 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:56.929508    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"
Oct 30 02:57:57.946449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:57.946057    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"]
Oct 30 02:57:57.946449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:57.946317    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy" containerID="cri-o://3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318" gracePeriod=30
Oct 30 02:57:59.730744 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.730713    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"]
Oct 30 02:57:59.731370 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.731352    4278 cpu_manager.go:414] "containerMap: removing container" podUID="675d2806-7d9f-4b20-8138-943eb40781e2" containerName="revoke-keys"
Oct 30 02:57:59.731462 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.731372    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675d2806-7d9f-4b20-8138-943eb40781e2" containerName="revoke-keys"
Oct 30 02:57:59.731519 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.731509    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="675d2806-7d9f-4b20-8138-943eb40781e2" containerName="revoke-keys"
Oct 30 02:57:59.735321 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.735300    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.757936 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.757913    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"]
Oct 30 02:57:59.767179 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.767156    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e11f03-openshift-default-dockercfg-ph97z\""
Oct 30 02:57:59.802937 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.802907    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4pm\" (UniqueName: \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-kube-api-access-nf4pm\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.803060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.802949    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.803060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.803018    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4bc478-f00d-435b-be38-c858acdc162c-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.803133 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.803100    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.803235 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.803221    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4bc478-f00d-435b-be38-c858acdc162c-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.803279 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.803266    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.803352 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.803339    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.803399 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.803369    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.803448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.803397    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.904582 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.904549    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4bc478-f00d-435b-be38-c858acdc162c-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.904746 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.904601    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.904746 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.904633    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.904746 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.904650    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.904746 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.904665    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.904746 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.904693    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4pm\" (UniqueName: \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-kube-api-access-nf4pm\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.904746 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.904724    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.905042 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.904752    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4bc478-f00d-435b-be38-c858acdc162c-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.905042 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.904776    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.905138 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.905074    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.905250 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.905218    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.905371 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.905229    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.905371 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.905262    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.905558 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.905540    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4bc478-f00d-435b-be38-c858acdc162c-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.907206 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.907184    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4bc478-f00d-435b-be38-c858acdc162c-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.907286 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.907236    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.924695 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.924673    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:57:59.924981 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:57:59.924964    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4pm\" (UniqueName: \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-kube-api-access-nf4pm\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:58:00.049002 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:00.048972    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:58:00.363172 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:58:00.363141    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4bc478_f00d_435b_be38_c858acdc162c.slice/crio-ca53f1083812e88e44edd34be42df742dbe7cdb6ee427d7f571738e81b4f0421 WatchSource:0}: Error finding container ca53f1083812e88e44edd34be42df742dbe7cdb6ee427d7f571738e81b4f0421: Status 404 returned error can't find the container with id ca53f1083812e88e44edd34be42df742dbe7cdb6ee427d7f571738e81b4f0421
Oct 30 02:58:00.363310 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:00.363207    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"]
Oct 30 02:58:00.366376 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:00.366212    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:58:00.368029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:00.367256    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:58:00.368029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:00.367388    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:58:01.058251 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:01.058209    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" event={"ID":"0b4bc478-f00d-435b-be38-c858acdc162c","Type":"ContainerStarted","Data":"2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f"}
Oct 30 02:58:01.058690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:01.058297    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" event={"ID":"0b4bc478-f00d-435b-be38-c858acdc162c","Type":"ContainerStarted","Data":"ca53f1083812e88e44edd34be42df742dbe7cdb6ee427d7f571738e81b4f0421"}
Oct 30 02:58:01.128446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:01.122819    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" podStartSLOduration=2.122801718 podStartE2EDuration="2.122801718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:01.116249244 +0000 UTC m=+1019.040457431" watchObservedRunningTime="2026-10-30 02:58:01.122801718 +0000 UTC m=+1019.047009904"
Oct 30 02:58:02.050096 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:02.050060    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:58:02.051826 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:02.051798    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:58:02.051937 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:02.051850    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 02:58:03.049979 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.049944    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:58:03.050371 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.050007    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 02:58:03.310193 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.310167    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:58:03.350320 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350291    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-credential-socket\") pod \"76ea4316-7b4f-4453-a930-3aea23f78fb1\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") "
Oct 30 02:58:03.350492 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350329    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-podinfo\") pod \"76ea4316-7b4f-4453-a930-3aea23f78fb1\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") "
Oct 30 02:58:03.350492 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350363    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-token\" (UniqueName: \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-token\") pod \"76ea4316-7b4f-4453-a930-3aea23f78fb1\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") "
Oct 30 02:58:03.350492 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350414    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-certs\") pod \"76ea4316-7b4f-4453-a930-3aea23f78fb1\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") "
Oct 30 02:58:03.350492 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350476    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-socket\") pod \"76ea4316-7b4f-4453-a930-3aea23f78fb1\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") "
Oct 30 02:58:03.350706 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350567    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-data\") pod \"76ea4316-7b4f-4453-a930-3aea23f78fb1\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") "
Oct 30 02:58:03.350706 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350592    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-envoy\") pod \"76ea4316-7b4f-4453-a930-3aea23f78fb1\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") "
Oct 30 02:58:03.350706 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350634    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-credential-socket" pod "76ea4316-7b4f-4453-a930-3aea23f78fb1" (UID: "76ea4316-7b4f-4453-a930-3aea23f78fb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:03.350706 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350680    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-kube-api-access-t84nq\" (UniqueName: \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-kube-api-access-t84nq\") pod \"76ea4316-7b4f-4453-a930-3aea23f78fb1\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") "
Oct 30 02:58:03.350922 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350805    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-certs" pod "76ea4316-7b4f-4453-a930-3aea23f78fb1" (UID: "76ea4316-7b4f-4453-a930-3aea23f78fb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:03.350922 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350856    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ea4316-7b4f-4453-a930-3aea23f78fb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ea4316-7b4f-4453-a930-3aea23f78fb1-istiod-ca-cert\") pod \"76ea4316-7b4f-4453-a930-3aea23f78fb1\" (UID: \"76ea4316-7b4f-4453-a930-3aea23f78fb1\") "
Oct 30 02:58:03.350922 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350891    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-data" pod "76ea4316-7b4f-4453-a930-3aea23f78fb1" (UID: "76ea4316-7b4f-4453-a930-3aea23f78fb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:03.350922 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.350910    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-socket" pod "76ea4316-7b4f-4453-a930-3aea23f78fb1" (UID: "76ea4316-7b4f-4453-a930-3aea23f78fb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:03.351272 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.351245    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ea4316-7b4f-4453-a930-3aea23f78fb1-istiod-ca-cert" pod "76ea4316-7b4f-4453-a930-3aea23f78fb1" (UID: "76ea4316-7b4f-4453-a930-3aea23f78fb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:03.351457 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.351413    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.351457 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.351450    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.351604 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.351467    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.351604 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.351480    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.352932 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.352908    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-envoy" pod "76ea4316-7b4f-4453-a930-3aea23f78fb1" (UID: "76ea4316-7b4f-4453-a930-3aea23f78fb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:03.353050 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.352953    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-podinfo" pod "76ea4316-7b4f-4453-a930-3aea23f78fb1" (UID: "76ea4316-7b4f-4453-a930-3aea23f78fb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 02:58:03.353050 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.352975    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-token" pod "76ea4316-7b4f-4453-a930-3aea23f78fb1" (UID: "76ea4316-7b4f-4453-a930-3aea23f78fb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:03.353050 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.353026    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-kube-api-access-t84nq" pod "76ea4316-7b4f-4453-a930-3aea23f78fb1" (UID: "76ea4316-7b4f-4453-a930-3aea23f78fb1"). InnerVolumeSpecName "kube-api-access-t84nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:03.453215 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.453180    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.453215 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.453209    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t84nq\" (UniqueName: \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-kube-api-access-t84nq\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.453215 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.453220    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ea4316-7b4f-4453-a930-3aea23f78fb1-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.453492 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.453229    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.453492 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:03.453238    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ea4316-7b4f-4453-a930-3aea23f78fb1-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:04.050210 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.050171    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:58:04.050597 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.050230    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 02:58:04.101384 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.101347    4278 generic.go:356] "Generic (PLEG): container finished" podID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerID="3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318" exitCode=0
Oct 30 02:58:04.101534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.101401    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"
Oct 30 02:58:04.101534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.101450    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" event={"ID":"76ea4316-7b4f-4453-a930-3aea23f78fb1","Type":"ContainerDied","Data":"3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318"}
Oct 30 02:58:04.101534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.101510    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw" event={"ID":"76ea4316-7b4f-4453-a930-3aea23f78fb1","Type":"ContainerDied","Data":"dc0a3dcba82ace132701d394a065f2d5e4efb20766b306ec3bae852a50a8a0d7"}
Oct 30 02:58:04.101657 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.101540    4278 scope.go:122] "RemoveContainer" containerID="3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318"
Oct 30 02:58:04.119893 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.119876    4278 scope.go:122] "RemoveContainer" containerID="26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790"
Oct 30 02:58:04.131543 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.131515    4278 scope.go:122] "RemoveContainer" containerID="3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318"
Oct 30 02:58:04.131914 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:58:04.131892    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318\": container with ID starting with 3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318 not found: ID does not exist" containerID="3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318"
Oct 30 02:58:04.132012 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.131925    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318"} err="failed to get container status \"3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318\": rpc error: code = NotFound desc = could not find container \"3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318\": container with ID starting with 3a7491f32b4d35355acdf8ac21ceb973bd1ec93c4c5e57d6c1f6b95f80d78318 not found: ID does not exist"
Oct 30 02:58:04.132012 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.131945    4278 scope.go:122] "RemoveContainer" containerID="26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790"
Oct 30 02:58:04.132273 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:58:04.132245    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790\": container with ID starting with 26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790 not found: ID does not exist" containerID="26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790"
Oct 30 02:58:04.132350 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.132283    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790"} err="failed to get container status \"26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790\": rpc error: code = NotFound desc = could not find container \"26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790\": container with ID starting with 26aa8000a9dfc840bd0a40adfabfe76634cd8d3d5b33efeb39f73325c2122790 not found: ID does not exist"
Oct 30 02:58:04.218941 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.218910    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"]
Oct 30 02:58:04.219104 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.218951    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w7qxw"]
Oct 30 02:58:04.627260 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.627229    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"]
Oct 30 02:58:04.627840 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.627821    4278 cpu_manager.go:414] "containerMap: removing container" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy"
Oct 30 02:58:04.627840 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.627840    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy"
Oct 30 02:58:04.627998 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.627848    4278 cpu_manager.go:414] "containerMap: removing container" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy"
Oct 30 02:58:04.627998 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.627853    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy"
Oct 30 02:58:04.627998 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.627951    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy"
Oct 30 02:58:04.627998 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.627959    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" containerName="istio-proxy"
Oct 30 02:58:04.633023 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.633004    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:04.648998 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.648974    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:04.648998 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.648976    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:04.649344 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.649327    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"]
Oct 30 02:58:04.693446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.691229    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5g54\" (UniqueName: \"kubernetes.io/projected/7c020155-58a3-4239-acb6-190c3edee0de-kube-api-access-q5g54\") pod \"maas-api-revoke-sub-0bc01355-pr67m\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:04.693446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.691481    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc01355-pr67m\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:04.693446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.691593    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc01355-pr67m\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:04.793291 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.793258    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc01355-pr67m\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:04.793476 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.793319    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5g54\" (UniqueName: \"kubernetes.io/projected/7c020155-58a3-4239-acb6-190c3edee0de-kube-api-access-q5g54\") pod \"maas-api-revoke-sub-0bc01355-pr67m\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:04.793548 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.793480    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc01355-pr67m\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:04.794237 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.794201    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc01355-pr67m\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:04.795777 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.795753    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc01355-pr67m\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:04.813152 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.813130    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5g54\" (UniqueName: \"kubernetes.io/projected/7c020155-58a3-4239-acb6-190c3edee0de-kube-api-access-q5g54\") pod \"maas-api-revoke-sub-0bc01355-pr67m\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:04.947195 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:04.947101    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:05.049842 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:05.049584    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:58:05.049842 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:05.049637    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 02:58:05.327712 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:05.327682    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"]
Oct 30 02:58:05.328078 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:58:05.327975    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c020155_58a3_4239_acb6_190c3edee0de.slice/crio-5b86c1b887b84ae3d2f3e55d95562b9a8dbfe1b9dc905706c4c7a192e6214b41 WatchSource:0}: Error finding container 5b86c1b887b84ae3d2f3e55d95562b9a8dbfe1b9dc905706c4c7a192e6214b41: Status 404 returned error can't find the container with id 5b86c1b887b84ae3d2f3e55d95562b9a8dbfe1b9dc905706c4c7a192e6214b41
Oct 30 02:58:06.050417 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:06.050384    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:58:06.050639 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:06.050471    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 02:58:06.120855 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:06.120819    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m" event={"ID":"7c020155-58a3-4239-acb6-190c3edee0de","Type":"ContainerStarted","Data":"f092b5ff7dfbb91b1c5223c6da9b0bc338b7bf0e5f8266e25d000ca42aff452b"}
Oct 30 02:58:06.121021 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:06.120887    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m" event={"ID":"7c020155-58a3-4239-acb6-190c3edee0de","Type":"ContainerStarted","Data":"5b86c1b887b84ae3d2f3e55d95562b9a8dbfe1b9dc905706c4c7a192e6214b41"}
Oct 30 02:58:06.170449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:06.166064    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m" podStartSLOduration=2.166045746 podStartE2EDuration="2.166045746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:06.162752297 +0000 UTC m=+1024.086960509" watchObservedRunningTime="2026-10-30 02:58:06.166045746 +0000 UTC m=+1024.090253934"
Oct 30 02:58:06.199980 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:06.199939    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ea4316-7b4f-4453-a930-3aea23f78fb1" path="/var/lib/kubelet/pods/76ea4316-7b4f-4453-a930-3aea23f78fb1/volumes"
Oct 30 02:58:07.129183 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:07.129152    4278 generic.go:356] "Generic (PLEG): container finished" podID="7c020155-58a3-4239-acb6-190c3edee0de" containerID="f092b5ff7dfbb91b1c5223c6da9b0bc338b7bf0e5f8266e25d000ca42aff452b" exitCode=0
Oct 30 02:58:07.129644 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:07.129237    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m" event={"ID":"7c020155-58a3-4239-acb6-190c3edee0de","Type":"ContainerDied","Data":"f092b5ff7dfbb91b1c5223c6da9b0bc338b7bf0e5f8266e25d000ca42aff452b"}
Oct 30 02:58:07.332110 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:07.332077    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:58:07.332597 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:07.332547    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:58:07.333420 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:07.333391    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 02:58:08.308715 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:08.308688    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:08.452953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:08.452862    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-xks-ca\") pod \"7c020155-58a3-4239-acb6-190c3edee0de\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") "
Oct 30 02:58:08.452953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:08.452922    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c020155-58a3-4239-acb6-190c3edee0de-kube-api-access-q5g54\" (UniqueName: \"kubernetes.io/projected/7c020155-58a3-4239-acb6-190c3edee0de-kube-api-access-q5g54\") pod \"7c020155-58a3-4239-acb6-190c3edee0de\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") "
Oct 30 02:58:08.453183 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:08.452987    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-openshift-ca\") pod \"7c020155-58a3-4239-acb6-190c3edee0de\" (UID: \"7c020155-58a3-4239-acb6-190c3edee0de\") "
Oct 30 02:58:08.453377 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:08.453346    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-openshift-ca" pod "7c020155-58a3-4239-acb6-190c3edee0de" (UID: "7c020155-58a3-4239-acb6-190c3edee0de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:08.453544 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:08.453462    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:08.455027 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:08.455007    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-xks-ca" pod "7c020155-58a3-4239-acb6-190c3edee0de" (UID: "7c020155-58a3-4239-acb6-190c3edee0de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:08.455101 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:08.455080    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c020155-58a3-4239-acb6-190c3edee0de-kube-api-access-q5g54" pod "7c020155-58a3-4239-acb6-190c3edee0de" (UID: "7c020155-58a3-4239-acb6-190c3edee0de"). InnerVolumeSpecName "kube-api-access-q5g54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:08.554992 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:08.554964    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c020155-58a3-4239-acb6-190c3edee0de-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:08.554992 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:08.554988    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5g54\" (UniqueName: \"kubernetes.io/projected/7c020155-58a3-4239-acb6-190c3edee0de-kube-api-access-q5g54\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:09.143471 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:09.143435    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"
Oct 30 02:58:09.143471 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:09.143457    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m" event={"ID":"7c020155-58a3-4239-acb6-190c3edee0de","Type":"ContainerDied","Data":"5b86c1b887b84ae3d2f3e55d95562b9a8dbfe1b9dc905706c4c7a192e6214b41"}
Oct 30 02:58:09.143693 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:09.143505    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b86c1b887b84ae3d2f3e55d95562b9a8dbfe1b9dc905706c4c7a192e6214b41"
Oct 30 02:58:12.488168 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.488134    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"]
Oct 30 02:58:12.488721 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.488707    4278 cpu_manager.go:414] "containerMap: removing container" podUID="7c020155-58a3-4239-acb6-190c3edee0de" containerName="revoke-keys"
Oct 30 02:58:12.488721 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.488722    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c020155-58a3-4239-acb6-190c3edee0de" containerName="revoke-keys"
Oct 30 02:58:12.488836 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.488827    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c020155-58a3-4239-acb6-190c3edee0de" containerName="revoke-keys"
Oct 30 02:58:12.492407 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.492389    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:12.510380 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.510354    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"]
Oct 30 02:58:12.519641 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.519620    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:12.519641 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.519638    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:12.603467 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.603439    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee30a2a1-zhnd6\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:12.603612 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.603596    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk7p6\" (UniqueName: \"kubernetes.io/projected/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-kube-api-access-bk7p6\") pod \"maas-api-revoke-sub-ee30a2a1-zhnd6\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:12.603761 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.603740    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee30a2a1-zhnd6\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:12.705451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.705396    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee30a2a1-zhnd6\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:12.705576 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.705559    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk7p6\" (UniqueName: \"kubernetes.io/projected/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-kube-api-access-bk7p6\") pod \"maas-api-revoke-sub-ee30a2a1-zhnd6\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:12.705663 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.705646    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee30a2a1-zhnd6\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:12.706118 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.706090    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee30a2a1-zhnd6\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:12.708046 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.708025    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee30a2a1-zhnd6\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:12.745583 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.745532    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk7p6\" (UniqueName: \"kubernetes.io/projected/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-kube-api-access-bk7p6\") pod \"maas-api-revoke-sub-ee30a2a1-zhnd6\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:12.806319 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:12.806296    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:13.104945 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:13.104919    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"]
Oct 30 02:58:13.106871 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:58:13.106846    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb69b6c_d31d_41ed_b9b3_cc3e1f3515a5.slice/crio-cd92342e3a890edbc63a08b448f74cc848a92463b56854b08e363e25f970b63d WatchSource:0}: Error finding container cd92342e3a890edbc63a08b448f74cc848a92463b56854b08e363e25f970b63d: Status 404 returned error can't find the container with id cd92342e3a890edbc63a08b448f74cc848a92463b56854b08e363e25f970b63d
Oct 30 02:58:13.177797 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:13.177758    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6" event={"ID":"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5","Type":"ContainerStarted","Data":"cd92342e3a890edbc63a08b448f74cc848a92463b56854b08e363e25f970b63d"}
Oct 30 02:58:14.186175 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:14.186085    4278 generic.go:356] "Generic (PLEG): container finished" podID="5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5" containerID="ee2b0e4e0ebec593bf9c069d63399df8a19f7d5d1b54cef991f6ad57c7523024" exitCode=0
Oct 30 02:58:14.186561 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:14.186178    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6" event={"ID":"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5","Type":"ContainerDied","Data":"ee2b0e4e0ebec593bf9c069d63399df8a19f7d5d1b54cef991f6ad57c7523024"}
Oct 30 02:58:15.387202 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:15.387176    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:15.446823 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:15.446783    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-openshift-ca\") pod \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") "
Oct 30 02:58:15.447009 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:15.446839    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-xks-ca\") pod \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") "
Oct 30 02:58:15.447237 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:15.447199    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-openshift-ca" pod "5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5" (UID: "5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:15.447398 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:15.447368    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:15.448983 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:15.448960    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-xks-ca" pod "5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5" (UID: "5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:15.548653 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:15.548622    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-kube-api-access-bk7p6\" (UniqueName: \"kubernetes.io/projected/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-kube-api-access-bk7p6\") pod \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\" (UID: \"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5\") "
Oct 30 02:58:15.548973 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:15.548953    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:15.550735 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:15.550709    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-kube-api-access-bk7p6" pod "5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5" (UID: "5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5"). InnerVolumeSpecName "kube-api-access-bk7p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:15.650055 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:15.650027    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk7p6\" (UniqueName: \"kubernetes.io/projected/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5-kube-api-access-bk7p6\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:16.207259 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:16.207226    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6" event={"ID":"5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5","Type":"ContainerDied","Data":"cd92342e3a890edbc63a08b448f74cc848a92463b56854b08e363e25f970b63d"}
Oct 30 02:58:16.207399 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:16.207281    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd92342e3a890edbc63a08b448f74cc848a92463b56854b08e363e25f970b63d"
Oct 30 02:58:16.207399 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:16.207373    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"
Oct 30 02:58:18.850188 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:18.850157    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:58:21.432625 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:21.432570    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:21.450588 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:21.450552    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:21.563576 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:21.563529    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 02:58:21.620280 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:21.620239    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:21.723501 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:21.719495    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:58:22.075604 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:22.075560    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.166461 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:22.165539    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 02:58:22.559837 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:22.547940    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.729243 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:22.729199    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.730156 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:22.730132    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:58:22.857866 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:22.857764    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.975571 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:22.975529    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:22.975838 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:22.975807    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.107448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.084544    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:23.108344 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.108271    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:58:23.109613 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.109587    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:23.109783 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.109756    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.198334 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.198203    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 02:58:23.250655 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.250616    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:58:23.347594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.347549    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:23.347870 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.347840    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.439509 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.421857    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:23.470453 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.468865    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.526133 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.526082    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.527283 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.526370    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 02:58:23.780451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:23.773510    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.156500 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.152137    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.326189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.283317    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.326189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.306106    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:24.326189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.306174    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.377542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.377504    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:58:24.493916 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:58:24.493812    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:58:24.515453 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.499986    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.539558 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.539510    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.548958 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.548934    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:58:24.573446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.569269    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.573446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.569964    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:58:24.573446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.570783    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.600448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.592469    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:58:24.600723 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.600693    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.622441 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.622380    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.752118 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.749658    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:58:24.756216 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.756192    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:58:24.814539 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.814383    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.934798 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:24.934769    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:58:27.749839 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.749804    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"]
Oct 30 02:58:27.750452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.750414    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5" containerName="revoke-keys"
Oct 30 02:58:27.750452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.750447    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5" containerName="revoke-keys"
Oct 30 02:58:27.750567 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.750557    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5" containerName="revoke-keys"
Oct 30 02:58:27.753939 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.753924    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:27.771645 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.771621    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"]
Oct 30 02:58:27.780736 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.780711    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:27.780858 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.780800    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:27.931994 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.931959    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7fws\" (UniqueName: \"kubernetes.io/projected/fe67cad2-4e90-4721-8b5d-4c91538982c6-kube-api-access-k7fws\") pod \"maas-api-revoke-sub-290cb2f6-6nzfb\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:27.932153 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.932134    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-290cb2f6-6nzfb\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:27.932193 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:27.932182    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-290cb2f6-6nzfb\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:28.033448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.033390    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-290cb2f6-6nzfb\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:28.033605 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.033485    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-290cb2f6-6nzfb\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:28.033605 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.033532    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7fws\" (UniqueName: \"kubernetes.io/projected/fe67cad2-4e90-4721-8b5d-4c91538982c6-kube-api-access-k7fws\") pod \"maas-api-revoke-sub-290cb2f6-6nzfb\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:28.034166 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.034149    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-290cb2f6-6nzfb\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:28.035748 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.035727    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-290cb2f6-6nzfb\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:28.053717 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.053686    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7fws\" (UniqueName: \"kubernetes.io/projected/fe67cad2-4e90-4721-8b5d-4c91538982c6-kube-api-access-k7fws\") pod \"maas-api-revoke-sub-290cb2f6-6nzfb\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:28.068392 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.068370    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:28.364542 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.364511    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"]
Oct 30 02:58:28.365093 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:58:28.365061    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe67cad2_4e90_4721_8b5d_4c91538982c6.slice/crio-4397667dd7fa200e077491c1f32e59b5a3f8f82f5128a714bcd8a860edc97c8c WatchSource:0}: Error finding container 4397667dd7fa200e077491c1f32e59b5a3f8f82f5128a714bcd8a860edc97c8c: Status 404 returned error can't find the container with id 4397667dd7fa200e077491c1f32e59b5a3f8f82f5128a714bcd8a860edc97c8c
Oct 30 02:58:28.832256 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.832218    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb" event={"ID":"fe67cad2-4e90-4721-8b5d-4c91538982c6","Type":"ContainerStarted","Data":"1fab25380a8bbebeca990f3f0f568b33f863e76f52cd0c31b8cc79f3845a1ca3"}
Oct 30 02:58:28.832698 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.832317    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb" event={"ID":"fe67cad2-4e90-4721-8b5d-4c91538982c6","Type":"ContainerStarted","Data":"4397667dd7fa200e077491c1f32e59b5a3f8f82f5128a714bcd8a860edc97c8c"}
Oct 30 02:58:28.873875 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:28.873819    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb" podStartSLOduration=1.873804918 podStartE2EDuration="1.873804918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:28.872118868 +0000 UTC m=+1046.796327069" watchObservedRunningTime="2026-10-30 02:58:28.873804918 +0000 UTC m=+1046.798013104"
Oct 30 02:58:29.561377 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.561283    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"]
Oct 30 02:58:29.565991 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.565968    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.586463 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.586391    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"]
Oct 30 02:58:29.594118 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.594096    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d1788\"/\"distinct-2-w4-7d1788-kserve-self-signed-certs\""
Oct 30 02:58:29.755349 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.755311    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tmp-dir\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.755594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.755497    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-home\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.755594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.755551    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-dshm\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.755718 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.755653    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tls-certs\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.755879 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.755855    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6sz\" (UniqueName: \"kubernetes.io/projected/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-kube-api-access-vv6sz\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.756010 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.755893    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-model-cache\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.845362 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.845256    4278 generic.go:356] "Generic (PLEG): container finished" podID="fe67cad2-4e90-4721-8b5d-4c91538982c6" containerID="1fab25380a8bbebeca990f3f0f568b33f863e76f52cd0c31b8cc79f3845a1ca3" exitCode=0
Oct 30 02:58:29.845362 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.845338    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb" event={"ID":"fe67cad2-4e90-4721-8b5d-4c91538982c6","Type":"ContainerDied","Data":"1fab25380a8bbebeca990f3f0f568b33f863e76f52cd0c31b8cc79f3845a1ca3"}
Oct 30 02:58:29.857115 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.857080    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6sz\" (UniqueName: \"kubernetes.io/projected/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-kube-api-access-vv6sz\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.857278 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.857129    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-model-cache\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.857278 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.857179    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tmp-dir\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.857278 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.857269    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-home\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.857472 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.857317    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-dshm\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.857472 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.857348    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tls-certs\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.857645 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.857622    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-model-cache\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.857702 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.857657    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-home\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.858140 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.857897    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tmp-dir\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.860830 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.860790    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-dshm\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.861228 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.861188    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tls-certs\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.877713 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.877684    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6sz\" (UniqueName: \"kubernetes.io/projected/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-kube-api-access-vv6sz\") pod \"distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:29.883466 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:29.883438    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:30.228480 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:58:30.228450    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod544a7f7e_ee3c_4e2c_9da6_aa486a4e40b2.slice/crio-7901b09258e29351659f375d1da864c5776fde8649e757adc5ba71ead3385224 WatchSource:0}: Error finding container 7901b09258e29351659f375d1da864c5776fde8649e757adc5ba71ead3385224: Status 404 returned error can't find the container with id 7901b09258e29351659f375d1da864c5776fde8649e757adc5ba71ead3385224
Oct 30 02:58:30.230733 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:30.230708    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"]
Oct 30 02:58:30.853759 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:30.853726    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" event={"ID":"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2","Type":"ContainerStarted","Data":"8016ea993532bb9669f3975d3fa55f3c8b27135c6b5acf932a6e72e970933b12"}
Oct 30 02:58:30.854223 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:30.853767    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" event={"ID":"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2","Type":"ContainerStarted","Data":"7901b09258e29351659f375d1da864c5776fde8649e757adc5ba71ead3385224"}
Oct 30 02:58:30.883547 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:30.883502    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:30.896706 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:30.896651    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podStartSLOduration=1.896634389 podStartE2EDuration="1.896634389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:30.893774443 +0000 UTC m=+1048.817982629" watchObservedRunningTime="2026-10-30 02:58:30.896634389 +0000 UTC m=+1048.820842578"
Oct 30 02:58:30.998944 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:30.998923    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:31.178639 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.178563    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-xks-ca\") pod \"fe67cad2-4e90-4721-8b5d-4c91538982c6\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") "
Oct 30 02:58:31.178639 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.178630    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-openshift-ca\") pod \"fe67cad2-4e90-4721-8b5d-4c91538982c6\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") "
Oct 30 02:58:31.178834 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.178764    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe67cad2-4e90-4721-8b5d-4c91538982c6-kube-api-access-k7fws\" (UniqueName: \"kubernetes.io/projected/fe67cad2-4e90-4721-8b5d-4c91538982c6-kube-api-access-k7fws\") pod \"fe67cad2-4e90-4721-8b5d-4c91538982c6\" (UID: \"fe67cad2-4e90-4721-8b5d-4c91538982c6\") "
Oct 30 02:58:31.178962 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.178937    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-openshift-ca" pod "fe67cad2-4e90-4721-8b5d-4c91538982c6" (UID: "fe67cad2-4e90-4721-8b5d-4c91538982c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:31.179350 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.179329    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:31.180672 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.180645    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-xks-ca" pod "fe67cad2-4e90-4721-8b5d-4c91538982c6" (UID: "fe67cad2-4e90-4721-8b5d-4c91538982c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:31.180854 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.180836    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe67cad2-4e90-4721-8b5d-4c91538982c6-kube-api-access-k7fws" pod "fe67cad2-4e90-4721-8b5d-4c91538982c6" (UID: "fe67cad2-4e90-4721-8b5d-4c91538982c6"). InnerVolumeSpecName "kube-api-access-k7fws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:31.280985 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.280948    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe67cad2-4e90-4721-8b5d-4c91538982c6-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:31.280985 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.280983    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7fws\" (UniqueName: \"kubernetes.io/projected/fe67cad2-4e90-4721-8b5d-4c91538982c6-kube-api-access-k7fws\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:58:31.861858 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.861827    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"
Oct 30 02:58:31.862238 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.861853    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb" event={"ID":"fe67cad2-4e90-4721-8b5d-4c91538982c6","Type":"ContainerDied","Data":"4397667dd7fa200e077491c1f32e59b5a3f8f82f5128a714bcd8a860edc97c8c"}
Oct 30 02:58:31.862238 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:31.861914    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4397667dd7fa200e077491c1f32e59b5a3f8f82f5128a714bcd8a860edc97c8c"
Oct 30 02:58:39.884189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:39.884150    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:39.896771 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:39.896746    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:39.929855 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:39.929828    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:49.160451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.155323    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:49.554452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.534776    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.568343 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.567513    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 02:58:49.603498 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.602588    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.611454 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.607684    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:58:49.755478 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.741863    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.773480 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.759230    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.801488 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.789321    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 02:58:49.804391 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.804346    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.922028 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.921907    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 02:58:49.947569 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:49.947516    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.158703 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.156942    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.159451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.159407    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:58:50.628814 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.628771    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.630108 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.630084    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:58:50.644346 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.644302    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.645593 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.644619    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:58:50.646027 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.646006    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:58:50.646209 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.646195    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:58:50.708273 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.708217    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.708937 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.708909    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:58:50.819572 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.791701    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.968658 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.820198    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.012264 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.930558    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.012926 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.933370    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded"
Oct 30 02:58:51.020641 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.968139    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.020883 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:50.968906    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:58:51.021339 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.021318    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:58:51.021626 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.021603    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:51.211708 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.211679    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:58:51.380394 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.193527    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:51.384715 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.241888    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded"
Oct 30 02:58:51.415957 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.396484    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:58:51.445504 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.435650    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:58:51.513450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.447833    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 30 02:58:51.650880 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.650764    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.652327 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.652291    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.682491 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.682449    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.797314 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.795580    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.797314 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:51.795969    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.048131 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.048091    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 02:58:52.049207 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.049176    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.158494 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.151745    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 02:58:52.346899 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.346725    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:58:52.397451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.393532    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.405616 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.405578    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.420444 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.417861    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.458466 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.456977    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.471113 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.471039    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:58:52.473912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.473071    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:58:52.549186 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.549146    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.659211 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.659116    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.794756 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.794721    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:58:52.864298 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.864269    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:58:52.908013 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.905889    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:58:52.939514 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:52.939379    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:58:54.189505 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:58:54.189458    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:58:55.385795 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.385764    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"]
Oct 30 02:58:55.386404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.386382    4278 cpu_manager.go:414] "containerMap: removing container" podUID="fe67cad2-4e90-4721-8b5d-4c91538982c6" containerName="revoke-keys"
Oct 30 02:58:55.386404 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.386403    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe67cad2-4e90-4721-8b5d-4c91538982c6" containerName="revoke-keys"
Oct 30 02:58:55.386608 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.386532    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe67cad2-4e90-4721-8b5d-4c91538982c6" containerName="revoke-keys"
Oct 30 02:58:55.390066 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.390049    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.410043 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.409928    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"]
Oct 30 02:58:55.417102 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.417072    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"distinct-2-w6-32bad3-kserve-self-signed-certs\""
Oct 30 02:58:55.482325 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.482277    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bsv\" (UniqueName: \"kubernetes.io/projected/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-kube-api-access-k4bsv\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.482527 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.482355    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-dshm\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.482527 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.482465    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-home\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.482614 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.482583    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-model-cache\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.482698 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.482677    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tmp-dir\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.482817 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.482800    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tls-certs\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.583985 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.583955    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-dshm\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.584189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.583994    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-home\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.584189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.584040    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-model-cache\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.584189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.584090    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tmp-dir\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.584189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.584161    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tls-certs\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.584398 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.584267    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bsv\" (UniqueName: \"kubernetes.io/projected/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-kube-api-access-k4bsv\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.584852 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.584598    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-model-cache\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.584852 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.584631    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-home\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.584852 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.584747    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tmp-dir\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.587185 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.587160    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-dshm\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.587283 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.587211    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tls-certs\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.613445 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.612980    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bsv\" (UniqueName: \"kubernetes.io/projected/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-kube-api-access-k4bsv\") pod \"distinct-2-w6-32bad3-kserve-69856dd9-kmkfj\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:55.709559 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:55.709461    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:56.039477 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:56.039445    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"]
Oct 30 02:58:56.040755 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:58:56.040726    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb215a820_a1ac_49d0_9ad5_cd8a41b54d15.slice/crio-7c8977e9b184c15f58a23e59ab53aff8ce0a4b904148e2403a20bef6b0a1b1dc WatchSource:0}: Error finding container 7c8977e9b184c15f58a23e59ab53aff8ce0a4b904148e2403a20bef6b0a1b1dc: Status 404 returned error can't find the container with id 7c8977e9b184c15f58a23e59ab53aff8ce0a4b904148e2403a20bef6b0a1b1dc
Oct 30 02:58:56.362805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:56.362700    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" event={"ID":"b215a820-a1ac-49d0-9ad5-cd8a41b54d15","Type":"ContainerStarted","Data":"e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2"}
Oct 30 02:58:56.362805 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:56.362751    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" event={"ID":"b215a820-a1ac-49d0-9ad5-cd8a41b54d15","Type":"ContainerStarted","Data":"7c8977e9b184c15f58a23e59ab53aff8ce0a4b904148e2403a20bef6b0a1b1dc"}
Oct 30 02:58:56.405734 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:56.405677    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podStartSLOduration=1.405658512 podStartE2EDuration="1.405658512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:56.403544429 +0000 UTC m=+1074.327752617" watchObservedRunningTime="2026-10-30 02:58:56.405658512 +0000 UTC m=+1074.329866702"
Oct 30 02:58:56.710102 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:56.710010    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:58:58.951243 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:58.951207    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"]
Oct 30 02:58:58.956022 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:58.956001    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:58.973841 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:58.973808    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"]
Oct 30 02:58:58.983466 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:58.980704    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:58.983466 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:58.980867    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:59.025810 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.025770    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7e3054-4qbsv\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:59.025988 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.025963    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7e3054-4qbsv\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:59.026048 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.026018    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7m5l\" (UniqueName: \"kubernetes.io/projected/44e85a0d-7d83-4b15-a878-276aaeaa623b-kube-api-access-v7m5l\") pod \"maas-api-revoke-sub-7e7e3054-4qbsv\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:59.127186 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.127149    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7e3054-4qbsv\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:59.127397 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.127226    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7m5l\" (UniqueName: \"kubernetes.io/projected/44e85a0d-7d83-4b15-a878-276aaeaa623b-kube-api-access-v7m5l\") pod \"maas-api-revoke-sub-7e7e3054-4qbsv\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:59.127397 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.127368    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7e3054-4qbsv\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:59.127861 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.127842    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7e3054-4qbsv\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:59.129759 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.129734    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7e3054-4qbsv\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:59.147675 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.147647    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7m5l\" (UniqueName: \"kubernetes.io/projected/44e85a0d-7d83-4b15-a878-276aaeaa623b-kube-api-access-v7m5l\") pod \"maas-api-revoke-sub-7e7e3054-4qbsv\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:59.274977 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.274947    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:58:59.576402 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:58:59.576370    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e85a0d_7d83_4b15_a878_276aaeaa623b.slice/crio-a00f5edf1b6636f01dbc03d158dfc36605fa487854b92bedb0438c45361df57a WatchSource:0}: Error finding container a00f5edf1b6636f01dbc03d158dfc36605fa487854b92bedb0438c45361df57a: Status 404 returned error can't find the container with id a00f5edf1b6636f01dbc03d158dfc36605fa487854b92bedb0438c45361df57a
Oct 30 02:58:59.576740 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:58:59.576718    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"]
Oct 30 02:59:00.392355 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:00.392318    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv" event={"ID":"44e85a0d-7d83-4b15-a878-276aaeaa623b","Type":"ContainerStarted","Data":"b4397bc185bb8ebae4e92ad54360531c0d59dbe4adcd1df312f9ec37f8e098df"}
Oct 30 02:59:00.392787 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:00.392377    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv" event={"ID":"44e85a0d-7d83-4b15-a878-276aaeaa623b","Type":"ContainerStarted","Data":"a00f5edf1b6636f01dbc03d158dfc36605fa487854b92bedb0438c45361df57a"}
Oct 30 02:59:00.435677 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:00.435608    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv" podStartSLOduration=2.435589935 podStartE2EDuration="2.435589935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:00.432705071 +0000 UTC m=+1078.356913261" watchObservedRunningTime="2026-10-30 02:59:00.435589935 +0000 UTC m=+1078.359798124"
Oct 30 02:59:01.402092 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:01.402059    4278 generic.go:356] "Generic (PLEG): container finished" podID="44e85a0d-7d83-4b15-a878-276aaeaa623b" containerID="b4397bc185bb8ebae4e92ad54360531c0d59dbe4adcd1df312f9ec37f8e098df" exitCode=0
Oct 30 02:59:01.402535 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:01.402130    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv" event={"ID":"44e85a0d-7d83-4b15-a878-276aaeaa623b","Type":"ContainerDied","Data":"b4397bc185bb8ebae4e92ad54360531c0d59dbe4adcd1df312f9ec37f8e098df"}
Oct 30 02:59:02.574809 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:02.574779    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:59:02.676508 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:02.676380    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-openshift-ca\") pod \"44e85a0d-7d83-4b15-a878-276aaeaa623b\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") "
Oct 30 02:59:02.676688 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:02.676567    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-xks-ca\") pod \"44e85a0d-7d83-4b15-a878-276aaeaa623b\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") "
Oct 30 02:59:02.676688 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:02.676655    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e85a0d-7d83-4b15-a878-276aaeaa623b-kube-api-access-v7m5l\" (UniqueName: \"kubernetes.io/projected/44e85a0d-7d83-4b15-a878-276aaeaa623b-kube-api-access-v7m5l\") pod \"44e85a0d-7d83-4b15-a878-276aaeaa623b\" (UID: \"44e85a0d-7d83-4b15-a878-276aaeaa623b\") "
Oct 30 02:59:02.676870 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:02.676840    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-openshift-ca" pod "44e85a0d-7d83-4b15-a878-276aaeaa623b" (UID: "44e85a0d-7d83-4b15-a878-276aaeaa623b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:02.677060 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:02.677044    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:59:02.678841 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:02.678810    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-xks-ca" pod "44e85a0d-7d83-4b15-a878-276aaeaa623b" (UID: "44e85a0d-7d83-4b15-a878-276aaeaa623b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:59:02.678940 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:02.678868    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e85a0d-7d83-4b15-a878-276aaeaa623b-kube-api-access-v7m5l" pod "44e85a0d-7d83-4b15-a878-276aaeaa623b" (UID: "44e85a0d-7d83-4b15-a878-276aaeaa623b"). InnerVolumeSpecName "kube-api-access-v7m5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:02.778131 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:02.778095    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e85a0d-7d83-4b15-a878-276aaeaa623b-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:59:02.778131 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:02.778128    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7m5l\" (UniqueName: \"kubernetes.io/projected/44e85a0d-7d83-4b15-a878-276aaeaa623b-kube-api-access-v7m5l\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:59:03.417020 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:03.416985    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"
Oct 30 02:59:03.417209 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:03.417019    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv" event={"ID":"44e85a0d-7d83-4b15-a878-276aaeaa623b","Type":"ContainerDied","Data":"a00f5edf1b6636f01dbc03d158dfc36605fa487854b92bedb0438c45361df57a"}
Oct 30 02:59:03.417209 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:03.417064    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a00f5edf1b6636f01dbc03d158dfc36605fa487854b92bedb0438c45361df57a"
Oct 30 02:59:05.710164 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:05.710120    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:59:05.722711 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:05.722687    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:59:06.450047 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:06.450021    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:59:11.101076 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.101042    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"]
Oct 30 02:59:11.101676 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.101659    4278 cpu_manager.go:414] "containerMap: removing container" podUID="44e85a0d-7d83-4b15-a878-276aaeaa623b" containerName="revoke-keys"
Oct 30 02:59:11.101733 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.101691    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e85a0d-7d83-4b15-a878-276aaeaa623b" containerName="revoke-keys"
Oct 30 02:59:11.101822 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.101813    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e85a0d-7d83-4b15-a878-276aaeaa623b" containerName="revoke-keys"
Oct 30 02:59:11.105557 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.105538    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.124331 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.124307    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"]
Oct 30 02:59:11.133958 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.133933    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:59:11.134078 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.134005    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:59:11.187843 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.187812    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkpjf\" (UniqueName: \"kubernetes.io/projected/d47aa8be-a438-4c31-befa-1f5a98ddffc3-kube-api-access-pkpjf\") pod \"maas-api-revoke-sub-face5b93-jhttc\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.188018 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.187871    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-face5b93-jhttc\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.188189 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.188167    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-face5b93-jhttc\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.289660 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.289619    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-face5b93-jhttc\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.289912 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.289856    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkpjf\" (UniqueName: \"kubernetes.io/projected/d47aa8be-a438-4c31-befa-1f5a98ddffc3-kube-api-access-pkpjf\") pod \"maas-api-revoke-sub-face5b93-jhttc\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.290024 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.290002    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-face5b93-jhttc\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.290486 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.290462    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-face5b93-jhttc\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.292367 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.292349    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-face5b93-jhttc\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.312274 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.312250    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkpjf\" (UniqueName: \"kubernetes.io/projected/d47aa8be-a438-4c31-befa-1f5a98ddffc3-kube-api-access-pkpjf\") pod \"maas-api-revoke-sub-face5b93-jhttc\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.418602 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.418520    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:11.737759 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:11.737730    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"]
Oct 30 02:59:11.737878 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:59:11.737760    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47aa8be_a438_4c31_befa_1f5a98ddffc3.slice/crio-942beedfce126c43ce8aebf6aac0e7b93dffc5956782a32f0c8d0a88a7ed46d8 WatchSource:0}: Error finding container 942beedfce126c43ce8aebf6aac0e7b93dffc5956782a32f0c8d0a88a7ed46d8: Status 404 returned error can't find the container with id 942beedfce126c43ce8aebf6aac0e7b93dffc5956782a32f0c8d0a88a7ed46d8
Oct 30 02:59:12.480999 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:12.480912    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc" event={"ID":"d47aa8be-a438-4c31-befa-1f5a98ddffc3","Type":"ContainerStarted","Data":"ccca67314910d5b8b16a5fc89a85aa0fdbbb6c41593f86478cfd2c8596ce9d29"}
Oct 30 02:59:12.480999 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:12.480965    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc" event={"ID":"d47aa8be-a438-4c31-befa-1f5a98ddffc3","Type":"ContainerStarted","Data":"942beedfce126c43ce8aebf6aac0e7b93dffc5956782a32f0c8d0a88a7ed46d8"}
Oct 30 02:59:12.526660 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:12.526610    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc" podStartSLOduration=1.52659648 podStartE2EDuration="1.52659648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:12.524140219 +0000 UTC m=+1090.448348411" watchObservedRunningTime="2026-10-30 02:59:12.52659648 +0000 UTC m=+1090.450804706"
Oct 30 02:59:13.489191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:13.489155    4278 generic.go:356] "Generic (PLEG): container finished" podID="d47aa8be-a438-4c31-befa-1f5a98ddffc3" containerID="ccca67314910d5b8b16a5fc89a85aa0fdbbb6c41593f86478cfd2c8596ce9d29" exitCode=0
Oct 30 02:59:13.489625 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:13.489248    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc" event={"ID":"d47aa8be-a438-4c31-befa-1f5a98ddffc3","Type":"ContainerDied","Data":"ccca67314910d5b8b16a5fc89a85aa0fdbbb6c41593f86478cfd2c8596ce9d29"}
Oct 30 02:59:14.656184 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:14.656152    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:14.837835 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:14.837802    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-xks-ca\") pod \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") "
Oct 30 02:59:14.838021 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:14.837984    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-openshift-ca\") pod \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") "
Oct 30 02:59:14.838128 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:14.838111    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47aa8be-a438-4c31-befa-1f5a98ddffc3-kube-api-access-pkpjf\" (UniqueName: \"kubernetes.io/projected/d47aa8be-a438-4c31-befa-1f5a98ddffc3-kube-api-access-pkpjf\") pod \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\" (UID: \"d47aa8be-a438-4c31-befa-1f5a98ddffc3\") "
Oct 30 02:59:14.838386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:14.838365    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-openshift-ca" pod "d47aa8be-a438-4c31-befa-1f5a98ddffc3" (UID: "d47aa8be-a438-4c31-befa-1f5a98ddffc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:14.838637 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:14.838621    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:59:14.839960 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:14.839930    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-xks-ca" pod "d47aa8be-a438-4c31-befa-1f5a98ddffc3" (UID: "d47aa8be-a438-4c31-befa-1f5a98ddffc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:59:14.840099 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:14.840084    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47aa8be-a438-4c31-befa-1f5a98ddffc3-kube-api-access-pkpjf" pod "d47aa8be-a438-4c31-befa-1f5a98ddffc3" (UID: "d47aa8be-a438-4c31-befa-1f5a98ddffc3"). InnerVolumeSpecName "kube-api-access-pkpjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:14.940190 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:14.940157    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkpjf\" (UniqueName: \"kubernetes.io/projected/d47aa8be-a438-4c31-befa-1f5a98ddffc3-kube-api-access-pkpjf\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:59:14.940190 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:14.940189    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47aa8be-a438-4c31-befa-1f5a98ddffc3-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 02:59:15.137049 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.136973    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"]
Oct 30 02:59:15.137578 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.137563    4278 cpu_manager.go:414] "containerMap: removing container" podUID="d47aa8be-a438-4c31-befa-1f5a98ddffc3" containerName="revoke-keys"
Oct 30 02:59:15.137639 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.137578    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47aa8be-a438-4c31-befa-1f5a98ddffc3" containerName="revoke-keys"
Oct 30 02:59:15.137690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.137680    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47aa8be-a438-4c31-befa-1f5a98ddffc3" containerName="revoke-keys"
Oct 30 02:59:15.141562 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.141539    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.153178 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.153153    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"]
Oct 30 02:59:15.157147 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.157127    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.164881 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.164857    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"]
Oct 30 02:59:15.192053 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.192019    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"]
Oct 30 02:59:15.243175 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.243140    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-976hn\" (UniqueName: \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-kube-api-access-976hn\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.243343 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.243183    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.243343 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.243275    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.243456 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.243380    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.243456 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.243415    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.243551 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.243531    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.243594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.243558    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.243656 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.243630    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.243695 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.243669    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.345103 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345063    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.345270 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345115    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976hn\" (UniqueName: \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-kube-api-access-976hn\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.345270 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345190    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.345270 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345235    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.345415 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345274    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.345415 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345393    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.345587 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345445    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw2f2\" (UniqueName: \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-kube-api-access-rw2f2\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.345587 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345495    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.345690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345590    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.345690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345640    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.345690 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345680    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.345934 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345875    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.345934 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.345886    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.346046 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.346018    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.346046 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.346026    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.346153 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.346055    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.346153 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.346137    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.346267 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.346222    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.346332 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.346266    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.346332 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.346289    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6787da26-32e1-4016-8300-8d5f1a0143f5-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.346332 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.346304    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.346543 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.346415    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.346861 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.346838    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.348061 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.348040    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.348415 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.348391    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.365486 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.365455    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-976hn\" (UniqueName: \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-kube-api-access-976hn\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.365600 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.365577    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.447725 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.447647    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw2f2\" (UniqueName: \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-kube-api-access-rw2f2\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.447725 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.447717    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.447911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.447762    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.447911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.447810    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.447911 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.447835    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.448041 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.447932    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.448041 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.447974    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6787da26-32e1-4016-8300-8d5f1a0143f5-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.448141 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.448098    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.448218 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.448177    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.448285 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.448176    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.448285 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.448233    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.448394 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.448288    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.448491 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.448470    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.448856 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.448833    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6787da26-32e1-4016-8300-8d5f1a0143f5-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.450414 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.450389    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.450526 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.450508    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.458589 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.458568    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:15.470313 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.469079    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw2f2\" (UniqueName: \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-kube-api-access-rw2f2\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.473023 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.473002    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.478029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.478011    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:15.506696 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.506670    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"
Oct 30 02:59:15.506696 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.506691    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc" event={"ID":"d47aa8be-a438-4c31-befa-1f5a98ddffc3","Type":"ContainerDied","Data":"942beedfce126c43ce8aebf6aac0e7b93dffc5956782a32f0c8d0a88a7ed46d8"}
Oct 30 02:59:15.506932 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.506777    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="942beedfce126c43ce8aebf6aac0e7b93dffc5956782a32f0c8d0a88a7ed46d8"
Oct 30 02:59:15.793864 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:59:15.793836    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0ec525_1d16_4d6c_ab56_19a341bbc3cf.slice/crio-b4393e691f1a032a806bcc26a70c24ec6f6d24fb00fc816285ae31d26b4f499f WatchSource:0}: Error finding container b4393e691f1a032a806bcc26a70c24ec6f6d24fb00fc816285ae31d26b4f499f: Status 404 returned error can't find the container with id b4393e691f1a032a806bcc26a70c24ec6f6d24fb00fc816285ae31d26b4f499f
Oct 30 02:59:15.795305 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.795278    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"]
Oct 30 02:59:15.796665 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.796629    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.796747 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.796713    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.796787 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.796753    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.808070 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.808045    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"]
Oct 30 02:59:15.811310 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:59:15.811284    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6787da26_32e1_4016_8300_8d5f1a0143f5.slice/crio-232ca9cbcb569e4533a6181842572a70b77b7c9c633edfb5544b273f9edd3af3 WatchSource:0}: Error finding container 232ca9cbcb569e4533a6181842572a70b77b7c9c633edfb5544b273f9edd3af3: Status 404 returned error can't find the container with id 232ca9cbcb569e4533a6181842572a70b77b7c9c633edfb5544b273f9edd3af3
Oct 30 02:59:15.815021 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.814983    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.815105 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.815059    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.815105 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.815093    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.925727 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:15.925688    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 02:59:16.529929 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:16.529687    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" event={"ID":"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf","Type":"ContainerStarted","Data":"4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549"}
Oct 30 02:59:16.529929 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:16.529820    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" event={"ID":"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf","Type":"ContainerStarted","Data":"b4393e691f1a032a806bcc26a70c24ec6f6d24fb00fc816285ae31d26b4f499f"}
Oct 30 02:59:16.532931 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:16.532891    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" event={"ID":"6787da26-32e1-4016-8300-8d5f1a0143f5","Type":"ContainerStarted","Data":"0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181"}
Oct 30 02:59:16.533117 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:16.532969    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" event={"ID":"6787da26-32e1-4016-8300-8d5f1a0143f5","Type":"ContainerStarted","Data":"232ca9cbcb569e4533a6181842572a70b77b7c9c633edfb5544b273f9edd3af3"}
Oct 30 02:59:16.655532 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:16.655470    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podStartSLOduration=1.655454721 podStartE2EDuration="1.655454721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:16.652507527 +0000 UTC m=+1094.576715712" watchObservedRunningTime="2026-10-30 02:59:16.655454721 +0000 UTC m=+1094.579662909"
Oct 30 02:59:16.656172 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:16.656137    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podStartSLOduration=1.656126212 podStartE2EDuration="1.656126212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:16.596915225 +0000 UTC m=+1094.521123414" watchObservedRunningTime="2026-10-30 02:59:16.656126212 +0000 UTC m=+1094.580334399"
Oct 30 02:59:17.468327 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:17.468291    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:17.469246 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:17.469220    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:17.469479 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:17.469452    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:17.481025 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:17.480495    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:17.489494 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:17.486638    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:17.489494 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:17.486715    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:18.207629 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.207582    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.406451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.403172    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.411546 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.411507    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 02:59:18.460075 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.459973    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:18.460388 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.460357    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:18.489450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.484545    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:18.489450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.484600    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:18.625647 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.625587    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.634818 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.634780    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.703642 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.703578    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.799638 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.799577    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.881638 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.881581    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.929623 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:18.929572    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.059466 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.058521    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.059466 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.058962    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 02:59:19.209450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.208533    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.209450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.209114    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:59:19.430272 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.430168    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.445619 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.445587    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:59:19.522452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.491368    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:19.522452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.491465    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:19.685558 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.656526    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 30 02:59:19.685558 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.657016    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:59:19.704587 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.704531    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.736449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.708929    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:59:19.804597 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.804548    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.850683 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.837362    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:59:19.876489 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.872979    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 02:59:19.909453 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.898247    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:19.933505 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.929669    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:59:19.941506 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.937785    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:19.941506 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.937848    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:19.941506 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.939836    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.968464 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:19.968013    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:59:20.067574 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.067528    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.108448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.082655    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:59:20.295479 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.067828    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.313463 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.209193    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 02:59:20.403556 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.403508    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.552519 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.446869    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.581577 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.467124    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.597034 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.470312    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:59:20.619451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.493164    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:20.623592 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.623547    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:20.631556 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.535560    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:20.713614 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.713556    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:20.719736 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.712071    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 30 02:59:20.748451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.712031    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.859792 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.859674    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 02:59:20.881534 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.881482    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.882149 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.882124    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:59:21.046931 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.946483    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.050803 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.962506    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded"
Oct 30 02:59:21.051013 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:20.962694    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.086458 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.070698    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.268455 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.259146    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded"
Oct 30 02:59:21.268455 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.259258    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.309028 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.301517    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.316457 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.315591    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.418452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.416787    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:59:21.418452 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.417556    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:59:21.464110 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.464078    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:21.464374 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.464346    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:21.522504 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.497266    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.542014 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.530942    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:21.542357 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.542312    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:21.623521 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.623468    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.624508 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.624474    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.624953 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.624934    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:59:21.750622 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.750560    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.793553 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.779068    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.869470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.861231    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 02:59:21.941451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:21.907210    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.055161 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.053503    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 30 02:59:22.055161 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.053873    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:22.072255 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.070900    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.298374 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.298323    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:22.298681 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.298645    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.303541 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.303507    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.317613 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.317517    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.417596 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.417550    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.426588 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.426510    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.460620 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.460569    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.462111 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.462074    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:22.462327 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.462305    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:22.482363 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.482330    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:22.482663 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.482637    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:22.542177 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.542118    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.633194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.629498    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.633194 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.629903    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 02:59:22.755358 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.755313    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.791009 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.790959    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.865334 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.865286    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.938089 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.937853    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:22.938351 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.938319    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.944142 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:22.944106    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.055560 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.055512    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.056178 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.056141    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 30 02:59:23.076470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.071666    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.078503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.077744    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.078503 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.077817    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.109531 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.102561    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.109531 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.102672    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.306470 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.304645    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.348188 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.348141    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 02:59:23.418022 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.417966    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.418338 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.418291    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.419949 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.419014    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.469592 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.469555    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:23.469909 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.469857    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:23.479826 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.479794    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:23.480084 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.480052    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:23.636510 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.630599    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 02:59:23.636510 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.631053    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 02:59:23.679449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.677856    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.679449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.677938    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.700603 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.700562    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.700897 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.700851    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.792447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.791583    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 30 02:59:23.792653 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.792495    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.916774 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.916673    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.943346 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.941822    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:59:23.948037 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:23.947505    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.087104 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.087065    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.087720 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.087695    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.106377 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.106349    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:59:24.232504 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.230152    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:24.232504 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.230227    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.320822 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.319488    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.362239 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.355581    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.434325 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:59:24.431768    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:59:24.434325 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.432084    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:59:24.434325 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.433283    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.460257 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.459813    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:24.460257 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.459873    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:24.500528 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.494115    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:24.500528 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.494169    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:24.643518 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.635523    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.806313 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.796037    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.806313 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.796442    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.956462 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:24.955672    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.147521 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.146511    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.147521 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.146908    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 02:59:25.342708 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.342662    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.356803 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.356750    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.450818 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.450711    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.459652 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.459622    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:25.460052 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.460002    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:25.479872 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.479836    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:25.480106 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.480077    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:25.636784 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.636739    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.695642 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.695575    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.798654 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.798591    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.799886 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.799829    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.858361 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.857488    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:59:25.996449 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:25.985754    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 02:59:26.039446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.039105    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:59:26.056505 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.056089    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 02:59:26.223408 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.223357    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 02:59:26.229622 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.229576    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:26.248917 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.248885    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 02:59:26.249103 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.248943    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 02:59:26.249103 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.248976    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 02:59:26.249594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.249570    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 02:59:26.249755 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.249744    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:59:26.249874 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.249863    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:59:26.514467 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.513403    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:26.514467 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.513487    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:26.519804 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.519767    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:26.520069 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:26.520037    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:27.478446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:27.470795    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:27.478446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:27.470860    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:27.490462 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:27.489798    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:27.490462 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:27.489861    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:28.459246 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:28.459213    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 02:59:28.459414 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:28.459279    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 02:59:28.481051 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:28.481015    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:59:28.481525 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:28.481082    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 02:59:29.725806 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:29.725765    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:29.726777 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:29.726708    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:29.727710 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:29.727691    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 02:59:30.478160 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:30.478119    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:30.519594 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:30.519564    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:30.519782 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:30.519649    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:31.022175 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:31.022128    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:31.484840 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:31.484754    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 02:59:37.114162 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.114125    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"]
Oct 30 02:59:37.143254 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.143222    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.148801 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.148705    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"]
Oct 30 02:59:37.190652 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.190621    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-socket\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.190822 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.190675    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-envoy\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.190822 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.190780    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-credential-socket\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.190937 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.190912    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ptcd\" (UniqueName: \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-kube-api-access-7ptcd\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.191001 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.190981    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-data\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.191141 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.191117    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-podinfo\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.191204 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.191173    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c464f3-8c2e-4996-b4c3-7361c529384d-istiod-ca-cert\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.191335 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.191313    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-certs\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.191519 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.191353    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-token\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.199989 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.199967    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-adca906b-openshift-default-dockercfg-6ghx8\""
Oct 30 02:59:37.292661 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.292631    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-podinfo\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.292834 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.292680    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c464f3-8c2e-4996-b4c3-7361c529384d-istiod-ca-cert\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.292834 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.292741    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-certs\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.292976 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.292937    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-token\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.293082 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.293061    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-socket\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.293151 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.293126    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-envoy\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.293151 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.293137    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-certs\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.293260 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.293203    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-credential-socket\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.293318 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.293278    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ptcd\" (UniqueName: \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-kube-api-access-7ptcd\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.293375 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.293356    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-data\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.293567 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.293546    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-credential-socket\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.293654 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.293588    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c464f3-8c2e-4996-b4c3-7361c529384d-istiod-ca-cert\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.294090 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.294065    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-data\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.294192 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.294170    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-socket\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.295290 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.295269    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-envoy\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.295412 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.295387    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-podinfo\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.320386 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.320359    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-token\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.320570 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.320523    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ptcd\" (UniqueName: \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-kube-api-access-7ptcd\") pod \"e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") " pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.466498 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.463794    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:37.804853 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:59:37.804819    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c464f3_8c2e_4996_b4c3_7361c529384d.slice/crio-bd0ad495aebb5af0b2cccfb4d998634ce92324cfb7d7631d00111b48a7e7b392 WatchSource:0}: Error finding container bd0ad495aebb5af0b2cccfb4d998634ce92324cfb7d7631d00111b48a7e7b392: Status 404 returned error can't find the container with id bd0ad495aebb5af0b2cccfb4d998634ce92324cfb7d7631d00111b48a7e7b392
Oct 30 02:59:37.805685 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.805659    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"]
Oct 30 02:59:37.807527 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.807496    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:37.807609 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.807560    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:37.807609 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:37.807593    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:38.540836 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.540795    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" event={"ID":"89c464f3-8c2e-4996-b4c3-7361c529384d","Type":"ContainerStarted","Data":"f9a4436cc155887b6d8b4dd94768cb3c0e9dfc121e7247e9dd37a4c03c7e2f03"}
Oct 30 02:59:38.541326 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.540863    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" event={"ID":"89c464f3-8c2e-4996-b4c3-7361c529384d","Type":"ContainerStarted","Data":"bd0ad495aebb5af0b2cccfb4d998634ce92324cfb7d7631d00111b48a7e7b392"}
Oct 30 02:59:38.624701 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.624642    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" podStartSLOduration=1.6246232040000002 podStartE2EDuration="1.624623204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:38.621862513 +0000 UTC m=+1116.546070701" watchObservedRunningTime="2026-10-30 02:59:38.624623204 +0000 UTC m=+1116.548831393"
Oct 30 02:59:38.890775 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.890690    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"]
Oct 30 02:59:38.895474 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.895449    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 02:59:38.896335 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.896306    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"]
Oct 30 02:59:38.901677 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.901633    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 02:59:38.917527 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.917492    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e2b0506d\""
Oct 30 02:59:38.918357 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.918332    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"]
Oct 30 02:59:38.925927 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.925898    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"]
Oct 30 02:59:38.987653 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:38.987625    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e2b0506d-dockercfg-t49rc\""
Oct 30 02:59:39.030489 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.023545    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75485640-6686-42ef-b6fe-c9834e405540-plugins-config-volume\") pod \"payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk\" (UID: \"75485640-6686-42ef-b6fe-c9834e405540\") " pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 02:59:39.031169 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.031140    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw\" (UID: \"da5c5135-1f96-4fa4-b97b-b8fcd82eb187\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 02:59:39.031461 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.031408    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh7xq\" (UniqueName: \"kubernetes.io/projected/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-kube-api-access-rh7xq\") pod \"payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw\" (UID: \"da5c5135-1f96-4fa4-b97b-b8fcd82eb187\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 02:59:39.031863 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.031758    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bv5\" (UniqueName: \"kubernetes.io/projected/75485640-6686-42ef-b6fe-c9834e405540-kube-api-access-c9bv5\") pod \"payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk\" (UID: \"75485640-6686-42ef-b6fe-c9834e405540\") " pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 02:59:39.183523 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.182656    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75485640-6686-42ef-b6fe-c9834e405540-plugins-config-volume\") pod \"payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk\" (UID: \"75485640-6686-42ef-b6fe-c9834e405540\") " pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 02:59:39.183523 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.182733    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw\" (UID: \"da5c5135-1f96-4fa4-b97b-b8fcd82eb187\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 02:59:39.183523 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.182879    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh7xq\" (UniqueName: \"kubernetes.io/projected/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-kube-api-access-rh7xq\") pod \"payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw\" (UID: \"da5c5135-1f96-4fa4-b97b-b8fcd82eb187\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 02:59:39.183523 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.183181    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bv5\" (UniqueName: \"kubernetes.io/projected/75485640-6686-42ef-b6fe-c9834e405540-kube-api-access-c9bv5\") pod \"payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk\" (UID: \"75485640-6686-42ef-b6fe-c9834e405540\") " pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 02:59:39.184439 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.184396    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw\" (UID: \"da5c5135-1f96-4fa4-b97b-b8fcd82eb187\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 02:59:39.185043 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.184981    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75485640-6686-42ef-b6fe-c9834e405540-plugins-config-volume\") pod \"payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk\" (UID: \"75485640-6686-42ef-b6fe-c9834e405540\") " pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 02:59:39.203732 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.203701    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bv5\" (UniqueName: \"kubernetes.io/projected/75485640-6686-42ef-b6fe-c9834e405540-kube-api-access-c9bv5\") pod \"payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk\" (UID: \"75485640-6686-42ef-b6fe-c9834e405540\") " pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 02:59:39.203867 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.203843    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh7xq\" (UniqueName: \"kubernetes.io/projected/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-kube-api-access-rh7xq\") pod \"payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw\" (UID: \"da5c5135-1f96-4fa4-b97b-b8fcd82eb187\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 02:59:39.218575 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.218551    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 02:59:39.235609 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.235542    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 02:59:39.464387 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.464315    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:39.469292 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.469264    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:39.548810 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.548779    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:39.549716 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.549698    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 02:59:39.557732 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:59:39.557707    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5c5135_1f96_4fa4_b97b_b8fcd82eb187.slice/crio-b7b53f84a350ae3c6d3c597f87c15649d3c9e3e8d2ed77ea790d2408a33dc16d WatchSource:0}: Error finding container b7b53f84a350ae3c6d3c597f87c15649d3c9e3e8d2ed77ea790d2408a33dc16d: Status 404 returned error can't find the container with id b7b53f84a350ae3c6d3c597f87c15649d3c9e3e8d2ed77ea790d2408a33dc16d
Oct 30 02:59:39.558635 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.558617    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"]
Oct 30 02:59:39.594366 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:39.594343    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"]
Oct 30 02:59:40.557382 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:40.557342    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk" event={"ID":"75485640-6686-42ef-b6fe-c9834e405540","Type":"ContainerStarted","Data":"36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a"}
Oct 30 02:59:40.557848 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:40.557406    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk" event={"ID":"75485640-6686-42ef-b6fe-c9834e405540","Type":"ContainerStarted","Data":"8efde8df27a40f375331b056c235d3605691438b7bae16fe2d602cc2d351f44a"}
Oct 30 02:59:40.557848 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:40.557539    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 02:59:40.560472 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:40.559471    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw" event={"ID":"da5c5135-1f96-4fa4-b97b-b8fcd82eb187","Type":"ContainerStarted","Data":"46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd"}
Oct 30 02:59:40.560472 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:40.559522    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw" event={"ID":"da5c5135-1f96-4fa4-b97b-b8fcd82eb187","Type":"ContainerStarted","Data":"b7b53f84a350ae3c6d3c597f87c15649d3c9e3e8d2ed77ea790d2408a33dc16d"}
Oct 30 02:59:40.560472 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:40.559650    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 02:59:40.613077 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:40.613021    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk" podStartSLOduration=2.613007999 podStartE2EDuration="2.613007999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:40.609256299 +0000 UTC m=+1118.533464487" watchObservedRunningTime="2026-10-30 02:59:40.613007999 +0000 UTC m=+1118.537216187"
Oct 30 02:59:40.660378 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:40.660324    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw" podStartSLOduration=2.660307968 podStartE2EDuration="2.660307968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:40.657397254 +0000 UTC m=+1118.581605442" watchObservedRunningTime="2026-10-30 02:59:40.660307968 +0000 UTC m=+1118.584516156"
Oct 30 02:59:40.948313 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:40.948213    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"]
Oct 30 02:59:41.007314 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.007285    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 02:59:41.012288 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.012253    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"]
Oct 30 02:59:41.027270 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.026916    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-93fe9de7\""
Oct 30 02:59:41.028674 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.028610    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-93fe9de7-dockercfg-p5f5d\""
Oct 30 02:59:41.137916 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.137878    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dee307-b518-4950-93b3-c99f2551ddce-plugins-config-volume\") pod \"payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4\" (UID: \"f4dee307-b518-4950-93b3-c99f2551ddce\") " pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 02:59:41.138188 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.138000    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7b8\" (UniqueName: \"kubernetes.io/projected/f4dee307-b518-4950-93b3-c99f2551ddce-kube-api-access-hk7b8\") pod \"payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4\" (UID: \"f4dee307-b518-4950-93b3-c99f2551ddce\") " pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 02:59:41.239322 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.239226    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dee307-b518-4950-93b3-c99f2551ddce-plugins-config-volume\") pod \"payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4\" (UID: \"f4dee307-b518-4950-93b3-c99f2551ddce\") " pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 02:59:41.239499 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.239367    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7b8\" (UniqueName: \"kubernetes.io/projected/f4dee307-b518-4950-93b3-c99f2551ddce-kube-api-access-hk7b8\") pod \"payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4\" (UID: \"f4dee307-b518-4950-93b3-c99f2551ddce\") " pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 02:59:41.239970 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.239949    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dee307-b518-4950-93b3-c99f2551ddce-plugins-config-volume\") pod \"payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4\" (UID: \"f4dee307-b518-4950-93b3-c99f2551ddce\") " pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 02:59:41.263684 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.263651    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7b8\" (UniqueName: \"kubernetes.io/projected/f4dee307-b518-4950-93b3-c99f2551ddce-kube-api-access-hk7b8\") pod \"payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4\" (UID: \"f4dee307-b518-4950-93b3-c99f2551ddce\") " pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 02:59:41.335904 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.335872    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 02:59:41.642135 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:59:41.642103    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dee307_b518_4950_93b3_c99f2551ddce.slice/crio-8a84d2d03a9631530cf0e168eefd33d827c6c151adfcfd6aaff08d2731aac409 WatchSource:0}: Error finding container 8a84d2d03a9631530cf0e168eefd33d827c6c151adfcfd6aaff08d2731aac409: Status 404 returned error can't find the container with id 8a84d2d03a9631530cf0e168eefd33d827c6c151adfcfd6aaff08d2731aac409
Oct 30 02:59:41.642818 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:41.642795    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"]
Oct 30 02:59:42.575493 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:42.575446    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4" event={"ID":"f4dee307-b518-4950-93b3-c99f2551ddce","Type":"ContainerStarted","Data":"fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba"}
Oct 30 02:59:42.575668 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:42.575515    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4" event={"ID":"f4dee307-b518-4950-93b3-c99f2551ddce","Type":"ContainerStarted","Data":"8a84d2d03a9631530cf0e168eefd33d827c6c151adfcfd6aaff08d2731aac409"}
Oct 30 02:59:42.575668 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:42.575654    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 02:59:42.631864 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:42.631792    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4" podStartSLOduration=2.631773632 podStartE2EDuration="2.631773632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:42.627191943 +0000 UTC m=+1120.551400134" watchObservedRunningTime="2026-10-30 02:59:42.631773632 +0000 UTC m=+1120.555981822"
Oct 30 02:59:42.964598 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:42.947519    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"]
Oct 30 02:59:42.990272 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:42.989797    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"]
Oct 30 02:59:42.990272 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:42.989895    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 02:59:43.011490 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:43.009735    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-adca906b\""
Oct 30 02:59:43.011490 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:43.009764    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-adca906b-dockercfg-gg64l\""
Oct 30 02:59:43.042127 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:43.042098    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v56ws\" (UniqueName: \"kubernetes.io/projected/74ad723e-7a76-4fe1-985e-8cd0149b7f60-kube-api-access-v56ws\") pod \"payload-processing-e2e-ait-adca906b-565c86788b-klv42\" (UID: \"74ad723e-7a76-4fe1-985e-8cd0149b7f60\") " pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 02:59:43.042278 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:43.042205    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ad723e-7a76-4fe1-985e-8cd0149b7f60-plugins-config-volume\") pod \"payload-processing-e2e-ait-adca906b-565c86788b-klv42\" (UID: \"74ad723e-7a76-4fe1-985e-8cd0149b7f60\") " pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 02:59:43.143672 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:43.143634    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v56ws\" (UniqueName: \"kubernetes.io/projected/74ad723e-7a76-4fe1-985e-8cd0149b7f60-kube-api-access-v56ws\") pod \"payload-processing-e2e-ait-adca906b-565c86788b-klv42\" (UID: \"74ad723e-7a76-4fe1-985e-8cd0149b7f60\") " pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 02:59:43.143858 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:43.143711    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ad723e-7a76-4fe1-985e-8cd0149b7f60-plugins-config-volume\") pod \"payload-processing-e2e-ait-adca906b-565c86788b-klv42\" (UID: \"74ad723e-7a76-4fe1-985e-8cd0149b7f60\") " pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 02:59:43.144442 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:43.144408    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ad723e-7a76-4fe1-985e-8cd0149b7f60-plugins-config-volume\") pod \"payload-processing-e2e-ait-adca906b-565c86788b-klv42\" (UID: \"74ad723e-7a76-4fe1-985e-8cd0149b7f60\") " pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 02:59:43.164406 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:43.164376    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v56ws\" (UniqueName: \"kubernetes.io/projected/74ad723e-7a76-4fe1-985e-8cd0149b7f60-kube-api-access-v56ws\") pod \"payload-processing-e2e-ait-adca906b-565c86788b-klv42\" (UID: \"74ad723e-7a76-4fe1-985e-8cd0149b7f60\") " pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 02:59:43.307243 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:43.307217    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 02:59:43.610636 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:43.610609    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"]
Oct 30 02:59:43.611024 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:59:43.611002    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ad723e_7a76_4fe1_985e_8cd0149b7f60.slice/crio-b627f5b52c155fd6940b104e90e190214136a0c8a6f98d83049fa5bdbac334b2 WatchSource:0}: Error finding container b627f5b52c155fd6940b104e90e190214136a0c8a6f98d83049fa5bdbac334b2: Status 404 returned error can't find the container with id b627f5b52c155fd6940b104e90e190214136a0c8a6f98d83049fa5bdbac334b2
Oct 30 02:59:44.469683 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.469639    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"]
Oct 30 02:59:44.485708 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.485671    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.499463 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.499407    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"]
Oct 30 02:59:44.563491 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.563456    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.563679 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.563526    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.563679 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.563646    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44332434-cfc7-4809-a3f5-4cb038a2a794-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.563795 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.563678    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wndrz\" (UniqueName: \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-kube-api-access-wndrz\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.563795 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.563773    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.563917 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.563831    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.563917 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.563907    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.564029 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.563941    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.564111 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.564091    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.591389 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.591355    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42" event={"ID":"74ad723e-7a76-4fe1-985e-8cd0149b7f60","Type":"ContainerStarted","Data":"b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2"}
Oct 30 02:59:44.591555 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.591409    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42" event={"ID":"74ad723e-7a76-4fe1-985e-8cd0149b7f60","Type":"ContainerStarted","Data":"b627f5b52c155fd6940b104e90e190214136a0c8a6f98d83049fa5bdbac334b2"}
Oct 30 02:59:44.591555 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.591464    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 02:59:44.654974 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.654920    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42" podStartSLOduration=2.654903238 podStartE2EDuration="2.654903238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:44.652468057 +0000 UTC m=+1122.576676245" watchObservedRunningTime="2026-10-30 02:59:44.654903238 +0000 UTC m=+1122.579111428"
Oct 30 02:59:44.665558 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.665534    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.665689 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.665594    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.665689 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.665669    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44332434-cfc7-4809-a3f5-4cb038a2a794-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.665809 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.665693    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndrz\" (UniqueName: \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-kube-api-access-wndrz\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.665947 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.665866    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.665947 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.665926    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.666066 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.665978    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.666066 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.666012    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.666066 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.666042    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.666214 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.666123    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.666214 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.666133    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.666309 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.666223    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.666409 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.666393    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.666704 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.666684    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44332434-cfc7-4809-a3f5-4cb038a2a794-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.670072 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.670049    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.670191 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.670174    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.693699 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.693670    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wndrz\" (UniqueName: \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-kube-api-access-wndrz\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.695765 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.695719    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:44.809668 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:44.808438    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:45.190902 ip-10-0-128-214 kubenswrapper[4278]: W1030 02:59:45.190873    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44332434_cfc7_4809_a3f5_4cb038a2a794.slice/crio-dd0979b75018525380ed006a627a68ba2842a8f954a544a22e3e05372ce7f4b2 WatchSource:0}: Error finding container dd0979b75018525380ed006a627a68ba2842a8f954a544a22e3e05372ce7f4b2: Status 404 returned error can't find the container with id dd0979b75018525380ed006a627a68ba2842a8f954a544a22e3e05372ce7f4b2
Oct 30 02:59:45.191777 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:45.191748    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"]
Oct 30 02:59:45.193463 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:45.193417    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:45.193529 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:45.193497    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:45.193567 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:45.193528    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:45.608293 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:45.604510    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" event={"ID":"44332434-cfc7-4809-a3f5-4cb038a2a794","Type":"ContainerStarted","Data":"eff671dc579ecf252b1f9440ca427baf353415b198c2e73bf9f4ca298b6a911b"}
Oct 30 02:59:45.608293 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:45.604573    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" event={"ID":"44332434-cfc7-4809-a3f5-4cb038a2a794","Type":"ContainerStarted","Data":"dd0979b75018525380ed006a627a68ba2842a8f954a544a22e3e05372ce7f4b2"}
Oct 30 02:59:45.676584 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:45.676534    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podStartSLOduration=1.6764559289999998 podStartE2EDuration="1.676455929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:45.662131355 +0000 UTC m=+1123.586339543" watchObservedRunningTime="2026-10-30 02:59:45.676455929 +0000 UTC m=+1123.600664114"
Oct 30 02:59:45.809282 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:45.809251    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:46.811137 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:46.811102    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 02:59:46.811623 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:46.811165    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 02:59:47.809955 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:47.809920    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 02:59:47.810162 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:47.809989    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 02:59:48.813450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:48.809987    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 02:59:48.813450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:48.810079    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 02:59:49.809792 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:49.809753    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 02:59:49.809979 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:49.809831    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 02:59:50.809341 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:50.809307    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 02:59:50.809824 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:50.809366    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 02:59:51.572450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:51.571565    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 02:59:51.583468 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:51.572879    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 02:59:51.947502 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:51.913588    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 02:59:52.074553 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:52.074505    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 02:59:52.635608 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:52.635553    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:52.743941 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:52.743897    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:52.788900 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:52.787513    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:52.871669 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:52.871632    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 02:59:52.871960 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:52.871926    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 02:59:52.914666 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:52.914559    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:52.965467 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:52.964379    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 02:59:53.088451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.064971    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:53.111443 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.079552    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.111443 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.107580    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.112132 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.112098    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.145287 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.145255    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:53.145559 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.145530    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.154446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.152549    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.327456 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.321576    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 02:59:53.367643 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.367556    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.424557 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.424509    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.575794 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.574719    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:53.576126 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.576074    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.585684 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.585567    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 02:59:53.613684 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.613647    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 02:59:53.645562 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.645514    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.646225 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.646205    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:59:53.915760 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.915647    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 02:59:53.921320 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.921143    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:59:54.026457 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:53.808395    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.030447 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.027060    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 02:59:54.140443 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.122008    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:54.140443 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.123035    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.140443 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.124317    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:59:54.140443 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.124404    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:59:54.168497 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.165009    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.179451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.179393    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:59:54.515507 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.502977    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:59:54.528497 ip-10-0-128-214 kubenswrapper[4278]: E1030 02:59:54.522711    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 02:59:54.647833 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.647792    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.687501 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.687471    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:59:54.750765 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.750706    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.864510 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.861865    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:54.864510 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.861933    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.935451 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.927106    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.996448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:54.993845    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.035461 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.024866    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.149512 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.140653    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.150134 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.150095    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:55.177320 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.177282    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.336450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.336231    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 02:59:55.340448 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.337331    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.452450 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.437103    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 02:59:55.504361 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.504298    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.596821 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.596779    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.637929 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.635614    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 02:59:55.664844 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.664804    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 02:59:55.729611 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.729515    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.790609 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.790564    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.904618 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.904584    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:55.904919 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:55.904872    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:56.043770 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.043725    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:56.092843 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.092043    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 02:59:56.182446 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.180783    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:56.183165 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.183145    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:56.184785 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.184768    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 02:59:56.186569 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.186094    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 02:59:56.195597 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.190292    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 30 02:59:56.364808 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.364732    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 02:59:56.377592 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.377567    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 02:59:56.396631 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.396402    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 02:59:56.440846 ip-10-0-128-214 kubenswrapper[4278]: I1030 02:59:56.439123    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 03:00:00.157576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.157540    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"]
Oct 30 03:00:00.162820 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.162767    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"
Oct 30 03:00:00.163953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.163924    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"]
Oct 30 03:00:00.191670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.191635    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"]
Oct 30 03:00:00.192083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.192054    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"
Oct 30 03:00:00.192539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.192518    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:00:00.198827 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.198768    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"
Oct 30 03:00:00.276063 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.276031    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"]
Oct 30 03:00:00.283075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.283044    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"]
Oct 30 03:00:00.284010 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.283988    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"
Oct 30 03:00:00.298783 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.298751    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"]
Oct 30 03:00:00.298926 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.298872    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"
Oct 30 03:00:00.305498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.304976    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"]
Oct 30 03:00:00.327072 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.327039    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dchkq\" (UniqueName: \"kubernetes.io/projected/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-kube-api-access-dchkq\") pod \"maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr\" (UID: \"5e34a96e-72d0-4c46-b988-d12cbfdb8a00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"
Oct 30 03:00:00.327320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.327294    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f2b109e-89e0-4616-bb5f-0be53618f755-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9\" (UID: \"5f2b109e-89e0-4616-bb5f-0be53618f755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"
Oct 30 03:00:00.327619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.327601    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9\" (UID: \"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"
Oct 30 03:00:00.327790 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.327771    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p29fc\" (UniqueName: \"kubernetes.io/projected/5f2b109e-89e0-4616-bb5f-0be53618f755-kube-api-access-p29fc\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9\" (UID: \"5f2b109e-89e0-4616-bb5f-0be53618f755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"
Oct 30 03:00:00.327899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.327883    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp942\" (UniqueName: \"kubernetes.io/projected/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-kube-api-access-lp942\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9\" (UID: \"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"
Oct 30 03:00:00.328017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.328002    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr\" (UID: \"5e34a96e-72d0-4c46-b988-d12cbfdb8a00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"
Oct 30 03:00:00.331510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.331491    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"]
Oct 30 03:00:00.338439 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.337336    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"]
Oct 30 03:00:00.343444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.343407    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"]
Oct 30 03:00:00.429079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.428996    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v\" (UID: \"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"
Oct 30 03:00:00.429079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429048    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f2b109e-89e0-4616-bb5f-0be53618f755-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9\" (UID: \"5f2b109e-89e0-4616-bb5f-0be53618f755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"
Oct 30 03:00:00.429278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429131    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjml\" (UniqueName: \"kubernetes.io/projected/eda28edd-6a3f-425f-8f62-c6109e64bb5e-kube-api-access-4kjml\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm\" (UID: \"eda28edd-6a3f-425f-8f62-c6109e64bb5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"
Oct 30 03:00:00.429278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429189    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2tp\" (UniqueName: \"kubernetes.io/projected/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-kube-api-access-dz2tp\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v\" (UID: \"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"
Oct 30 03:00:00.429402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429324    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9\" (UID: \"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"
Oct 30 03:00:00.429510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429409    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p29fc\" (UniqueName: \"kubernetes.io/projected/5f2b109e-89e0-4616-bb5f-0be53618f755-kube-api-access-p29fc\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9\" (UID: \"5f2b109e-89e0-4616-bb5f-0be53618f755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"
Oct 30 03:00:00.429510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429487    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp942\" (UniqueName: \"kubernetes.io/projected/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-kube-api-access-lp942\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9\" (UID: \"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"
Oct 30 03:00:00.429618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429531    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr\" (UID: \"5e34a96e-72d0-4c46-b988-d12cbfdb8a00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"
Oct 30 03:00:00.429679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429648    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eda28edd-6a3f-425f-8f62-c6109e64bb5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm\" (UID: \"eda28edd-6a3f-425f-8f62-c6109e64bb5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"
Oct 30 03:00:00.429815 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429791    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dchkq\" (UniqueName: \"kubernetes.io/projected/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-kube-api-access-dchkq\") pod \"maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr\" (UID: \"5e34a96e-72d0-4c46-b988-d12cbfdb8a00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"
Oct 30 03:00:00.429965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429931    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f2b109e-89e0-4616-bb5f-0be53618f755-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9\" (UID: \"5f2b109e-89e0-4616-bb5f-0be53618f755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"
Oct 30 03:00:00.430086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.429979    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9\" (UID: \"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"
Oct 30 03:00:00.430300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.430280    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr\" (UID: \"5e34a96e-72d0-4c46-b988-d12cbfdb8a00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"
Oct 30 03:00:00.449990 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.449965    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p29fc\" (UniqueName: \"kubernetes.io/projected/5f2b109e-89e0-4616-bb5f-0be53618f755-kube-api-access-p29fc\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9\" (UID: \"5f2b109e-89e0-4616-bb5f-0be53618f755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"
Oct 30 03:00:00.450370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.450352    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dchkq\" (UniqueName: \"kubernetes.io/projected/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-kube-api-access-dchkq\") pod \"maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr\" (UID: \"5e34a96e-72d0-4c46-b988-d12cbfdb8a00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"
Oct 30 03:00:00.450545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.450524    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp942\" (UniqueName: \"kubernetes.io/projected/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-kube-api-access-lp942\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9\" (UID: \"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"
Oct 30 03:00:00.502772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.502737    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"
Oct 30 03:00:00.519367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.519335    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"
Oct 30 03:00:00.533060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.533025    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eda28edd-6a3f-425f-8f62-c6109e64bb5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm\" (UID: \"eda28edd-6a3f-425f-8f62-c6109e64bb5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"
Oct 30 03:00:00.533369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.533341    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v\" (UID: \"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"
Oct 30 03:00:00.533493 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.533471    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjml\" (UniqueName: \"kubernetes.io/projected/eda28edd-6a3f-425f-8f62-c6109e64bb5e-kube-api-access-4kjml\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm\" (UID: \"eda28edd-6a3f-425f-8f62-c6109e64bb5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"
Oct 30 03:00:00.533554 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.533545    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2tp\" (UniqueName: \"kubernetes.io/projected/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-kube-api-access-dz2tp\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v\" (UID: \"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"
Oct 30 03:00:00.534753 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.534731    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eda28edd-6a3f-425f-8f62-c6109e64bb5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm\" (UID: \"eda28edd-6a3f-425f-8f62-c6109e64bb5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"
Oct 30 03:00:00.535406 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.535378    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v\" (UID: \"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"
Oct 30 03:00:00.554561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.554533    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2tp\" (UniqueName: \"kubernetes.io/projected/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-kube-api-access-dz2tp\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v\" (UID: \"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"
Oct 30 03:00:00.558681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.558648    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjml\" (UniqueName: \"kubernetes.io/projected/eda28edd-6a3f-425f-8f62-c6109e64bb5e-kube-api-access-4kjml\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm\" (UID: \"eda28edd-6a3f-425f-8f62-c6109e64bb5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"
Oct 30 03:00:00.568029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.567995    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"
Oct 30 03:00:00.637002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.636971    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"
Oct 30 03:00:00.643690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.641602    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"
Oct 30 03:00:00.843389 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:00.843339    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e34a96e_72d0_4c46_b988_d12cbfdb8a00.slice/crio-5915afb4a9e1e89d76a53f27331de2512aef773d262cbe84a2eafbb5fab9f637 WatchSource:0}: Error finding container 5915afb4a9e1e89d76a53f27331de2512aef773d262cbe84a2eafbb5fab9f637: Status 404 returned error can't find the container with id 5915afb4a9e1e89d76a53f27331de2512aef773d262cbe84a2eafbb5fab9f637
Oct 30 03:00:00.844783 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.844751    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"]
Oct 30 03:00:00.858040 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.858005    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"]
Oct 30 03:00:00.860616 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:00.860587    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab69754_6a43_4d6e_8e36_ba706d7fe4d9.slice/crio-07a3bdbb78cd89b744b1432d3cd879dd51d01f06731cb1d7b6608f9ca20e60f5 WatchSource:0}: Error finding container 07a3bdbb78cd89b744b1432d3cd879dd51d01f06731cb1d7b6608f9ca20e60f5: Status 404 returned error can't find the container with id 07a3bdbb78cd89b744b1432d3cd879dd51d01f06731cb1d7b6608f9ca20e60f5
Oct 30 03:00:00.895932 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:00.895904    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2b109e_89e0_4616_bb5f_0be53618f755.slice/crio-91b9a979de9cd859be8c61ffdccf56590e6204fcb05d1f995fa2509d2b96e55c WatchSource:0}: Error finding container 91b9a979de9cd859be8c61ffdccf56590e6204fcb05d1f995fa2509d2b96e55c: Status 404 returned error can't find the container with id 91b9a979de9cd859be8c61ffdccf56590e6204fcb05d1f995fa2509d2b96e55c
Oct 30 03:00:00.896506 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.896374    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"]
Oct 30 03:00:00.976736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:00.976708    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"]
Oct 30 03:00:01.007545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.007516    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"]
Oct 30 03:00:01.562036 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.561765    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" event={"ID":"5e34a96e-72d0-4c46-b988-d12cbfdb8a00","Type":"ContainerStarted","Data":"616db2054822d4e6981eeb3d5c86071a468d2044c6f0435ccef91296b28d3d28"}
Oct 30 03:00:01.562036 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.561835    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" event={"ID":"5e34a96e-72d0-4c46-b988-d12cbfdb8a00","Type":"ContainerStarted","Data":"5915afb4a9e1e89d76a53f27331de2512aef773d262cbe84a2eafbb5fab9f637"}
Oct 30 03:00:01.563608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.563580    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" event={"ID":"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03","Type":"ContainerStarted","Data":"61dc84d6cbd0541687bd26bc57269dda0574bd7a6a35fbfc91a030ac3677719a"}
Oct 30 03:00:01.565079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.565055    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" event={"ID":"eda28edd-6a3f-425f-8f62-c6109e64bb5e","Type":"ContainerStarted","Data":"cd0f4e2cb30edc7303c4a08f68d020a10b3927ea1b45f807232502577c685ddc"}
Oct 30 03:00:01.566893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.566869    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" event={"ID":"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9","Type":"ContainerStarted","Data":"f6e2a79bc1c81df83ac2d3f91ef140b9636dc781a7d261f2869be0878e71094f"}
Oct 30 03:00:01.567086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.567061    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" event={"ID":"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9","Type":"ContainerStarted","Data":"07a3bdbb78cd89b744b1432d3cd879dd51d01f06731cb1d7b6608f9ca20e60f5"}
Oct 30 03:00:01.568813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.568792    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" event={"ID":"5f2b109e-89e0-4616-bb5f-0be53618f755","Type":"ContainerStarted","Data":"73a9290f0aefd4d4643bad6482b89e41ef12a05f9f19a1581987e43e4b9ba23b"}
Oct 30 03:00:01.568920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.568827    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" event={"ID":"5f2b109e-89e0-4616-bb5f-0be53618f755","Type":"ContainerStarted","Data":"91b9a979de9cd859be8c61ffdccf56590e6204fcb05d1f995fa2509d2b96e55c"}
Oct 30 03:00:01.606017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.605961    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" podStartSLOduration=1.605946814 podStartE2EDuration="1.605946814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:01.602944707 +0000 UTC m=+1139.527152895" watchObservedRunningTime="2026-10-30 03:00:01.605946814 +0000 UTC m=+1139.530155003"
Oct 30 03:00:01.646541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.646490    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" podStartSLOduration=1.6464761540000001 podStartE2EDuration="1.646476154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:01.643069172 +0000 UTC m=+1139.567277361" watchObservedRunningTime="2026-10-30 03:00:01.646476154 +0000 UTC m=+1139.570684375"
Oct 30 03:00:01.689592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:01.689547    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" podStartSLOduration=1.689535037 podStartE2EDuration="1.689535037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:01.685454828 +0000 UTC m=+1139.609663020" watchObservedRunningTime="2026-10-30 03:00:01.689535037 +0000 UTC m=+1139.613743225"
Oct 30 03:00:02.587215 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.587119    4278 generic.go:356] "Generic (PLEG): container finished" podID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerID="616db2054822d4e6981eeb3d5c86071a468d2044c6f0435ccef91296b28d3d28" exitCode=7
Oct 30 03:00:02.587744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.587211    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" event={"ID":"5e34a96e-72d0-4c46-b988-d12cbfdb8a00","Type":"ContainerDied","Data":"616db2054822d4e6981eeb3d5c86071a468d2044c6f0435ccef91296b28d3d28"}
Oct 30 03:00:02.587744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.587587    4278 scope.go:122] "RemoveContainer" containerID="616db2054822d4e6981eeb3d5c86071a468d2044c6f0435ccef91296b28d3d28"
Oct 30 03:00:02.590258 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.590231    4278 generic.go:356] "Generic (PLEG): container finished" podID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerID="88f49956da6440a7a56a9c86c741677cffe43a4b133e097ee24954ec36c3acb0" exitCode=60
Oct 30 03:00:02.590344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.590270    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" event={"ID":"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03","Type":"ContainerDied","Data":"88f49956da6440a7a56a9c86c741677cffe43a4b133e097ee24954ec36c3acb0"}
Oct 30 03:00:02.590696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.590675    4278 scope.go:122] "RemoveContainer" containerID="88f49956da6440a7a56a9c86c741677cffe43a4b133e097ee24954ec36c3acb0"
Oct 30 03:00:02.592262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.592239    4278 generic.go:356] "Generic (PLEG): container finished" podID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerID="a113e9ba440305a6df6f5a42d6163dbdf667179b9c19746e9e39b889fc5fba41" exitCode=60
Oct 30 03:00:02.592363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.592331    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" event={"ID":"eda28edd-6a3f-425f-8f62-c6109e64bb5e","Type":"ContainerDied","Data":"a113e9ba440305a6df6f5a42d6163dbdf667179b9c19746e9e39b889fc5fba41"}
Oct 30 03:00:02.592698 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.592614    4278 scope.go:122] "RemoveContainer" containerID="a113e9ba440305a6df6f5a42d6163dbdf667179b9c19746e9e39b889fc5fba41"
Oct 30 03:00:02.594417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.594399    4278 generic.go:356] "Generic (PLEG): container finished" podID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerID="f6e2a79bc1c81df83ac2d3f91ef140b9636dc781a7d261f2869be0878e71094f" exitCode=60
Oct 30 03:00:02.594516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.594438    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" event={"ID":"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9","Type":"ContainerDied","Data":"f6e2a79bc1c81df83ac2d3f91ef140b9636dc781a7d261f2869be0878e71094f"}
Oct 30 03:00:02.594700 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.594685    4278 scope.go:122] "RemoveContainer" containerID="f6e2a79bc1c81df83ac2d3f91ef140b9636dc781a7d261f2869be0878e71094f"
Oct 30 03:00:02.596163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.596144    4278 generic.go:356] "Generic (PLEG): container finished" podID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerID="73a9290f0aefd4d4643bad6482b89e41ef12a05f9f19a1581987e43e4b9ba23b" exitCode=60
Oct 30 03:00:02.596245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.596223    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" event={"ID":"5f2b109e-89e0-4616-bb5f-0be53618f755","Type":"ContainerDied","Data":"73a9290f0aefd4d4643bad6482b89e41ef12a05f9f19a1581987e43e4b9ba23b"}
Oct 30 03:00:02.596382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:02.596369    4278 scope.go:122] "RemoveContainer" containerID="73a9290f0aefd4d4643bad6482b89e41ef12a05f9f19a1581987e43e4b9ba23b"
Oct 30 03:00:03.607822 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:03.607791    4278 generic.go:356] "Generic (PLEG): container finished" podID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerID="da334d1af5b1ebdf4ca748fc424e40eb35066750d67f9791ae3d876a6e77822a" exitCode=7
Oct 30 03:00:03.608375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:03.607891    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" event={"ID":"5e34a96e-72d0-4c46-b988-d12cbfdb8a00","Type":"ContainerDied","Data":"da334d1af5b1ebdf4ca748fc424e40eb35066750d67f9791ae3d876a6e77822a"}
Oct 30 03:00:03.608375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:03.607982    4278 scope.go:122] "RemoveContainer" containerID="616db2054822d4e6981eeb3d5c86071a468d2044c6f0435ccef91296b28d3d28"
Oct 30 03:00:03.608375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:03.608278    4278 scope.go:122] "RemoveContainer" containerID="da334d1af5b1ebdf4ca748fc424e40eb35066750d67f9791ae3d876a6e77822a"
Oct 30 03:00:03.609266 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:03.609240    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr_odh-ai-gateway-infra(5e34a96e-72d0-4c46-b988-d12cbfdb8a00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00"
Oct 30 03:00:03.635739 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:03.635705    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" event={"ID":"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03","Type":"ContainerStarted","Data":"df1e80f467f2dcb6ffe8701374f95d1453af0659d51bdbd11f765e516f707254"}
Oct 30 03:00:03.638036 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:03.638010    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" event={"ID":"eda28edd-6a3f-425f-8f62-c6109e64bb5e","Type":"ContainerStarted","Data":"4df0043d7cc9696f709a5a54b1d381b37ca9a71d2a23af1c35b7740783bd3600"}
Oct 30 03:00:03.640149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:03.640124    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" event={"ID":"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9","Type":"ContainerStarted","Data":"1380b4ec4ec08239eed0974684f0feecf55e57f6d5deca2c8eb735b6de99e35d"}
Oct 30 03:00:03.641967 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:03.641947    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" event={"ID":"5f2b109e-89e0-4616-bb5f-0be53618f755","Type":"ContainerStarted","Data":"8a8fc0cb0c941147c55ce5bbf6374ea9ecb4e3066ca0e062825c439bb4caa0b5"}
Oct 30 03:00:03.767393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:03.767335    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" podStartSLOduration=3.767320282 podStartE2EDuration="3.767320282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:03.766461597 +0000 UTC m=+1141.690669786" watchObservedRunningTime="2026-10-30 03:00:03.767320282 +0000 UTC m=+1141.691528470"
Oct 30 03:00:03.807405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:03.807337    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" podStartSLOduration=3.8073228009999998 podStartE2EDuration="3.807322801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:03.805655424 +0000 UTC m=+1141.729863613" watchObservedRunningTime="2026-10-30 03:00:03.807322801 +0000 UTC m=+1141.731530991"
Oct 30 03:00:04.038267 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.038231    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"]
Oct 30 03:00:04.044199 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.044164    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a966adec-jmfzd"]
Oct 30 03:00:04.202147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.202065    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bacea58-df0a-4ae3-8819-d709fb03645d" path="/var/lib/kubelet/pods/3bacea58-df0a-4ae3-8819-d709fb03645d/volumes"
Oct 30 03:00:04.651603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.651574    4278 scope.go:122] "RemoveContainer" containerID="da334d1af5b1ebdf4ca748fc424e40eb35066750d67f9791ae3d876a6e77822a"
Oct 30 03:00:04.652084 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:04.652025    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr_odh-ai-gateway-infra(5e34a96e-72d0-4c46-b988-d12cbfdb8a00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00"
Oct 30 03:00:04.653598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.653566    4278 generic.go:356] "Generic (PLEG): container finished" podID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerID="df1e80f467f2dcb6ffe8701374f95d1453af0659d51bdbd11f765e516f707254" exitCode=60
Oct 30 03:00:04.653757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.653668    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" event={"ID":"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03","Type":"ContainerDied","Data":"df1e80f467f2dcb6ffe8701374f95d1453af0659d51bdbd11f765e516f707254"}
Oct 30 03:00:04.653757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.653743    4278 scope.go:122] "RemoveContainer" containerID="88f49956da6440a7a56a9c86c741677cffe43a4b133e097ee24954ec36c3acb0"
Oct 30 03:00:04.653970 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.653950    4278 scope.go:122] "RemoveContainer" containerID="df1e80f467f2dcb6ffe8701374f95d1453af0659d51bdbd11f765e516f707254"
Oct 30 03:00:04.654266 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:04.654247    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v_odh-ai-gateway-infra(e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03"
Oct 30 03:00:04.656601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.656566    4278 generic.go:356] "Generic (PLEG): container finished" podID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerID="4df0043d7cc9696f709a5a54b1d381b37ca9a71d2a23af1c35b7740783bd3600" exitCode=60
Oct 30 03:00:04.656758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.656629    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" event={"ID":"eda28edd-6a3f-425f-8f62-c6109e64bb5e","Type":"ContainerDied","Data":"4df0043d7cc9696f709a5a54b1d381b37ca9a71d2a23af1c35b7740783bd3600"}
Oct 30 03:00:04.656868 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.656851    4278 scope.go:122] "RemoveContainer" containerID="4df0043d7cc9696f709a5a54b1d381b37ca9a71d2a23af1c35b7740783bd3600"
Oct 30 03:00:04.657206 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:04.657182    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm_odh-ai-gateway-infra(eda28edd-6a3f-425f-8f62-c6109e64bb5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e"
Oct 30 03:00:04.658838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.658806    4278 generic.go:356] "Generic (PLEG): container finished" podID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerID="1380b4ec4ec08239eed0974684f0feecf55e57f6d5deca2c8eb735b6de99e35d" exitCode=60
Oct 30 03:00:04.658956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.658933    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" event={"ID":"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9","Type":"ContainerDied","Data":"1380b4ec4ec08239eed0974684f0feecf55e57f6d5deca2c8eb735b6de99e35d"}
Oct 30 03:00:04.659089 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.659075    4278 scope.go:122] "RemoveContainer" containerID="1380b4ec4ec08239eed0974684f0feecf55e57f6d5deca2c8eb735b6de99e35d"
Oct 30 03:00:04.659352 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:04.659335    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9_odh-ai-gateway-infra(6ab69754-6a43-4d6e-8e36-ba706d7fe4d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9"
Oct 30 03:00:04.661143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.661118    4278 generic.go:356] "Generic (PLEG): container finished" podID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerID="8a8fc0cb0c941147c55ce5bbf6374ea9ecb4e3066ca0e062825c439bb4caa0b5" exitCode=60
Oct 30 03:00:04.661290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.661163    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" event={"ID":"5f2b109e-89e0-4616-bb5f-0be53618f755","Type":"ContainerDied","Data":"8a8fc0cb0c941147c55ce5bbf6374ea9ecb4e3066ca0e062825c439bb4caa0b5"}
Oct 30 03:00:04.661559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.661537    4278 scope.go:122] "RemoveContainer" containerID="8a8fc0cb0c941147c55ce5bbf6374ea9ecb4e3066ca0e062825c439bb4caa0b5"
Oct 30 03:00:04.661906 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:04.661885    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9_odh-ai-gateway-infra(5f2b109e-89e0-4616-bb5f-0be53618f755)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755"
Oct 30 03:00:04.675834 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.675797    4278 scope.go:122] "RemoveContainer" containerID="a113e9ba440305a6df6f5a42d6163dbdf667179b9c19746e9e39b889fc5fba41"
Oct 30 03:00:04.692694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.692665    4278 scope.go:122] "RemoveContainer" containerID="f6e2a79bc1c81df83ac2d3f91ef140b9636dc781a7d261f2869be0878e71094f"
Oct 30 03:00:04.708003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:04.707977    4278 scope.go:122] "RemoveContainer" containerID="73a9290f0aefd4d4643bad6482b89e41ef12a05f9f19a1581987e43e4b9ba23b"
Oct 30 03:00:05.670488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:05.670459    4278 scope.go:122] "RemoveContainer" containerID="df1e80f467f2dcb6ffe8701374f95d1453af0659d51bdbd11f765e516f707254"
Oct 30 03:00:05.670938 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:05.670856    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v_odh-ai-gateway-infra(e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03"
Oct 30 03:00:05.672032 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:05.672014    4278 scope.go:122] "RemoveContainer" containerID="4df0043d7cc9696f709a5a54b1d381b37ca9a71d2a23af1c35b7740783bd3600"
Oct 30 03:00:05.672349 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:05.672325    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm_odh-ai-gateway-infra(eda28edd-6a3f-425f-8f62-c6109e64bb5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e"
Oct 30 03:00:05.673537 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:05.673519    4278 scope.go:122] "RemoveContainer" containerID="1380b4ec4ec08239eed0974684f0feecf55e57f6d5deca2c8eb735b6de99e35d"
Oct 30 03:00:05.673827 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:05.673802    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9_odh-ai-gateway-infra(6ab69754-6a43-4d6e-8e36-ba706d7fe4d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9"
Oct 30 03:00:05.675059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:05.675044    4278 scope.go:122] "RemoveContainer" containerID="8a8fc0cb0c941147c55ce5bbf6374ea9ecb4e3066ca0e062825c439bb4caa0b5"
Oct 30 03:00:05.675306 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:05.675289    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9_odh-ai-gateway-infra(5f2b109e-89e0-4616-bb5f-0be53618f755)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755"
Oct 30 03:00:07.341761 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:07.341715    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:00:14.028975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.028939    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"]
Oct 30 03:00:14.042375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.042347    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2206a7-jgdrt"]
Oct 30 03:00:14.200787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.200756    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ed9263-f2a9-4c4c-9d15-8e786a4190ad" path="/var/lib/kubelet/pods/c2ed9263-f2a9-4c4c-9d15-8e786a4190ad/volumes"
Oct 30 03:00:14.590300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.590265    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"]
Oct 30 03:00:14.596897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.596870    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.620477 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.620445    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"]
Oct 30 03:00:14.679571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.679536    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.679571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.679568    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.679787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.679598    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.679787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.679696    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mrn\" (UniqueName: \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-kube-api-access-t2mrn\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.679787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.679779    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.679880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.679839    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.679990 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.679976    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.680033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.680022    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.680068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.680052    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.781626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.781592    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.781626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.781629    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.781867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.781663    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.781867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.781693    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mrn\" (UniqueName: \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-kube-api-access-t2mrn\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.781867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.781758    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.781867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.781835    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.782062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.781977    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.782062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.782041    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.782126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.782068    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.782182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.782150    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.782226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.782078    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.782361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.782335    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.782474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.782386    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.782634 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.782611    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.784000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.783979    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.784359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.784342    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.812659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.812635    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.812798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.812749    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mrn\" (UniqueName: \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-kube-api-access-t2mrn\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:14.916754 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:14.916666    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:15.192381 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.192294    4278 scope.go:122] "RemoveContainer" containerID="df1e80f467f2dcb6ffe8701374f95d1453af0659d51bdbd11f765e516f707254"
Oct 30 03:00:15.192792 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.192454    4278 scope.go:122] "RemoveContainer" containerID="8a8fc0cb0c941147c55ce5bbf6374ea9ecb4e3066ca0e062825c439bb4caa0b5"
Oct 30 03:00:15.192792 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.192506    4278 scope.go:122] "RemoveContainer" containerID="da334d1af5b1ebdf4ca748fc424e40eb35066750d67f9791ae3d876a6e77822a"
Oct 30 03:00:15.192792 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.192559    4278 scope.go:122] "RemoveContainer" containerID="1380b4ec4ec08239eed0974684f0feecf55e57f6d5deca2c8eb735b6de99e35d"
Oct 30 03:00:15.238529 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:15.238500    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2163c7f_a4d6_4648_9109_2c2cffa4ec37.slice/crio-faa3892e27d37e5a53eed27d6ca25d8c8f359863fade2e8f279cf433b3ca88bb WatchSource:0}: Error finding container faa3892e27d37e5a53eed27d6ca25d8c8f359863fade2e8f279cf433b3ca88bb: Status 404 returned error can't find the container with id faa3892e27d37e5a53eed27d6ca25d8c8f359863fade2e8f279cf433b3ca88bb
Oct 30 03:00:15.239508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.239480    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"]
Oct 30 03:00:15.241213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.241183    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:15.241295 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.241246    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:15.241295 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.241283    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:15.781361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.781329    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" event={"ID":"5e34a96e-72d0-4c46-b988-d12cbfdb8a00","Type":"ContainerStarted","Data":"17dff72c8a94aa7649c9d1b2afe13815e65b2d1dfc325c272a686f2a8364612f"}
Oct 30 03:00:15.783481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.783452    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" event={"ID":"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03","Type":"ContainerStarted","Data":"c6dd141b6128e8455e7a30eab9d137d889c086384a042ea7592d80eee476db14"}
Oct 30 03:00:15.785989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.785961    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" event={"ID":"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9","Type":"ContainerStarted","Data":"27969080717cac01489727a54f82dc2c8acd557b445f69e65aa59053d9fee0a8"}
Oct 30 03:00:15.789543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.789517    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" event={"ID":"e2163c7f-a4d6-4648-9109-2c2cffa4ec37","Type":"ContainerStarted","Data":"6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e"}
Oct 30 03:00:15.789657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.789567    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" event={"ID":"e2163c7f-a4d6-4648-9109-2c2cffa4ec37","Type":"ContainerStarted","Data":"faa3892e27d37e5a53eed27d6ca25d8c8f359863fade2e8f279cf433b3ca88bb"}
Oct 30 03:00:15.911165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.911115    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" podStartSLOduration=1.911100331 podStartE2EDuration="1.911100331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:15.907097483 +0000 UTC m=+1153.831305671" watchObservedRunningTime="2026-10-30 03:00:15.911100331 +0000 UTC m=+1153.835308519"
Oct 30 03:00:15.917366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:15.917322    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:16.798662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.798626    4278 generic.go:356] "Generic (PLEG): container finished" podID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerID="3df8a9d03495f122d0c3bb0f3667aebb0962f52d782ac8ab9f678e1400690c64" exitCode=60
Oct 30 03:00:16.799290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.798664    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" event={"ID":"5f2b109e-89e0-4616-bb5f-0be53618f755","Type":"ContainerDied","Data":"3df8a9d03495f122d0c3bb0f3667aebb0962f52d782ac8ab9f678e1400690c64"}
Oct 30 03:00:16.799290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.798736    4278 scope.go:122] "RemoveContainer" containerID="8a8fc0cb0c941147c55ce5bbf6374ea9ecb4e3066ca0e062825c439bb4caa0b5"
Oct 30 03:00:16.799802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.799783    4278 scope.go:122] "RemoveContainer" containerID="3df8a9d03495f122d0c3bb0f3667aebb0962f52d782ac8ab9f678e1400690c64"
Oct 30 03:00:16.800300 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:16.800280    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9_odh-ai-gateway-infra(5f2b109e-89e0-4616-bb5f-0be53618f755)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755"
Oct 30 03:00:16.801204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.801183    4278 generic.go:356] "Generic (PLEG): container finished" podID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerID="17dff72c8a94aa7649c9d1b2afe13815e65b2d1dfc325c272a686f2a8364612f" exitCode=60
Oct 30 03:00:16.801294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.801259    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" event={"ID":"5e34a96e-72d0-4c46-b988-d12cbfdb8a00","Type":"ContainerDied","Data":"17dff72c8a94aa7649c9d1b2afe13815e65b2d1dfc325c272a686f2a8364612f"}
Oct 30 03:00:16.801690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.801669    4278 scope.go:122] "RemoveContainer" containerID="17dff72c8a94aa7649c9d1b2afe13815e65b2d1dfc325c272a686f2a8364612f"
Oct 30 03:00:16.802026 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:16.802000    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr_odh-ai-gateway-infra(5e34a96e-72d0-4c46-b988-d12cbfdb8a00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00"
Oct 30 03:00:16.803022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.803002    4278 generic.go:356] "Generic (PLEG): container finished" podID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerID="c6dd141b6128e8455e7a30eab9d137d889c086384a042ea7592d80eee476db14" exitCode=60
Oct 30 03:00:16.803118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.803072    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" event={"ID":"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03","Type":"ContainerDied","Data":"c6dd141b6128e8455e7a30eab9d137d889c086384a042ea7592d80eee476db14"}
Oct 30 03:00:16.803489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.803468    4278 scope.go:122] "RemoveContainer" containerID="c6dd141b6128e8455e7a30eab9d137d889c086384a042ea7592d80eee476db14"
Oct 30 03:00:16.803939 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:16.803815    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v_odh-ai-gateway-infra(e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03"
Oct 30 03:00:16.805606 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.805587    4278 generic.go:356] "Generic (PLEG): container finished" podID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerID="27969080717cac01489727a54f82dc2c8acd557b445f69e65aa59053d9fee0a8" exitCode=60
Oct 30 03:00:16.805819 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.805799    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" event={"ID":"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9","Type":"ContainerDied","Data":"27969080717cac01489727a54f82dc2c8acd557b445f69e65aa59053d9fee0a8"}
Oct 30 03:00:16.806534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.806504    4278 scope.go:122] "RemoveContainer" containerID="27969080717cac01489727a54f82dc2c8acd557b445f69e65aa59053d9fee0a8"
Oct 30 03:00:16.806774 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:16.806728    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9_odh-ai-gateway-infra(6ab69754-6a43-4d6e-8e36-ba706d7fe4d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9"
Oct 30 03:00:16.813335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.813318    4278 scope.go:122] "RemoveContainer" containerID="da334d1af5b1ebdf4ca748fc424e40eb35066750d67f9791ae3d876a6e77822a"
Oct 30 03:00:16.826245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.826227    4278 scope.go:122] "RemoveContainer" containerID="df1e80f467f2dcb6ffe8701374f95d1453af0659d51bdbd11f765e516f707254"
Oct 30 03:00:16.837069 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.837045    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"]
Oct 30 03:00:16.840060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.840044    4278 scope.go:122] "RemoveContainer" containerID="1380b4ec4ec08239eed0974684f0feecf55e57f6d5deca2c8eb735b6de99e35d"
Oct 30 03:00:16.888990 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.888952    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"]
Oct 30 03:00:16.919871 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.919840    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 03:00:16.920011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.919902    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 03:00:16.983055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:16.983003    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"]
Oct 30 03:00:17.192765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:17.192671    4278 scope.go:122] "RemoveContainer" containerID="4df0043d7cc9696f709a5a54b1d381b37ca9a71d2a23af1c35b7740783bd3600"
Oct 30 03:00:17.817390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:17.817357    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" event={"ID":"eda28edd-6a3f-425f-8f62-c6109e64bb5e","Type":"ContainerStarted","Data":"a5de5fe915cfb1ef17b505cf1cebf6cacc4e56c0e5e8edb7ab67b7df8044e1ed"}
Oct 30 03:00:17.881461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:17.878153    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"]
Oct 30 03:00:17.923528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:17.920992    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 03:00:17.923528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:17.921047    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 03:00:18.109687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.109655    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"
Oct 30 03:00:18.242310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.242285    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-maas-api-service-ca\") pod \"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03\" (UID: \"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03\") "
Oct 30 03:00:18.242522 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.242502    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-kube-api-access-dz2tp\" (UniqueName: \"kubernetes.io/projected/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-kube-api-access-dz2tp\") pod \"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03\" (UID: \"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03\") "
Oct 30 03:00:18.242772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.242744    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-maas-api-service-ca" pod "e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" (UID: "e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:18.243218 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.243197    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:18.245216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.245193    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-kube-api-access-dz2tp" pod "e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" (UID: "e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03"). InnerVolumeSpecName "kube-api-access-dz2tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:18.247202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.247185    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"
Oct 30 03:00:18.289143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.289079    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"
Oct 30 03:00:18.293581 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.293562    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"
Oct 30 03:00:18.344291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.344264    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2b109e-89e0-4616-bb5f-0be53618f755-kube-api-access-p29fc\" (UniqueName: \"kubernetes.io/projected/5f2b109e-89e0-4616-bb5f-0be53618f755-kube-api-access-p29fc\") pod \"5f2b109e-89e0-4616-bb5f-0be53618f755\" (UID: \"5f2b109e-89e0-4616-bb5f-0be53618f755\") "
Oct 30 03:00:18.344462 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.344327    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-maas-api-service-ca\") pod \"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9\" (UID: \"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9\") "
Oct 30 03:00:18.344462 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.344459    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-kube-api-access-lp942\" (UniqueName: \"kubernetes.io/projected/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-kube-api-access-lp942\") pod \"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9\" (UID: \"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9\") "
Oct 30 03:00:18.344576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.344501    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-kube-api-access-dchkq\" (UniqueName: \"kubernetes.io/projected/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-kube-api-access-dchkq\") pod \"5e34a96e-72d0-4c46-b988-d12cbfdb8a00\" (UID: \"5e34a96e-72d0-4c46-b988-d12cbfdb8a00\") "
Oct 30 03:00:18.344576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.344552    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2b109e-89e0-4616-bb5f-0be53618f755-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f2b109e-89e0-4616-bb5f-0be53618f755-maas-api-service-ca\") pod \"5f2b109e-89e0-4616-bb5f-0be53618f755\" (UID: \"5f2b109e-89e0-4616-bb5f-0be53618f755\") "
Oct 30 03:00:18.344684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.344648    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-maas-api-service-ca" pod "6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" (UID: "6ab69754-6a43-4d6e-8e36-ba706d7fe4d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:18.344684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.344662    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-maas-api-service-ca\") pod \"5e34a96e-72d0-4c46-b988-d12cbfdb8a00\" (UID: \"5e34a96e-72d0-4c46-b988-d12cbfdb8a00\") "
Oct 30 03:00:18.345204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.345173    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2b109e-89e0-4616-bb5f-0be53618f755-maas-api-service-ca" pod "5f2b109e-89e0-4616-bb5f-0be53618f755" (UID: "5f2b109e-89e0-4616-bb5f-0be53618f755"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:18.345548 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.345528    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2tp\" (UniqueName: \"kubernetes.io/projected/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03-kube-api-access-dz2tp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:18.345548 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.345550    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:18.345733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.345561    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f2b109e-89e0-4616-bb5f-0be53618f755-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:18.345733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.345597    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-maas-api-service-ca" pod "5e34a96e-72d0-4c46-b988-d12cbfdb8a00" (UID: "5e34a96e-72d0-4c46-b988-d12cbfdb8a00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:18.347490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.347392    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2b109e-89e0-4616-bb5f-0be53618f755-kube-api-access-p29fc" pod "5f2b109e-89e0-4616-bb5f-0be53618f755" (UID: "5f2b109e-89e0-4616-bb5f-0be53618f755"). InnerVolumeSpecName "kube-api-access-p29fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:18.347490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.347467    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-kube-api-access-lp942" pod "6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" (UID: "6ab69754-6a43-4d6e-8e36-ba706d7fe4d9"). InnerVolumeSpecName "kube-api-access-lp942". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:18.347669 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.347501    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-kube-api-access-dchkq" pod "5e34a96e-72d0-4c46-b988-d12cbfdb8a00" (UID: "5e34a96e-72d0-4c46-b988-d12cbfdb8a00"). InnerVolumeSpecName "kube-api-access-dchkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:18.447834 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.447799    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:18.447834 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.447837    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p29fc\" (UniqueName: \"kubernetes.io/projected/5f2b109e-89e0-4616-bb5f-0be53618f755-kube-api-access-p29fc\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:18.448017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.447853    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp942\" (UniqueName: \"kubernetes.io/projected/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9-kube-api-access-lp942\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:18.448017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.447868    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dchkq\" (UniqueName: \"kubernetes.io/projected/5e34a96e-72d0-4c46-b988-d12cbfdb8a00-kube-api-access-dchkq\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:18.831589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.831558    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"
Oct 30 03:00:18.832032 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.831609    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9" event={"ID":"5f2b109e-89e0-4616-bb5f-0be53618f755","Type":"ContainerDied","Data":"91b9a979de9cd859be8c61ffdccf56590e6204fcb05d1f995fa2509d2b96e55c"}
Oct 30 03:00:18.832032 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.831669    4278 scope.go:122] "RemoveContainer" containerID="3df8a9d03495f122d0c3bb0f3667aebb0962f52d782ac8ab9f678e1400690c64"
Oct 30 03:00:18.833221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.833195    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"
Oct 30 03:00:18.833221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.833210    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr" event={"ID":"5e34a96e-72d0-4c46-b988-d12cbfdb8a00","Type":"ContainerDied","Data":"5915afb4a9e1e89d76a53f27331de2512aef773d262cbe84a2eafbb5fab9f637"}
Oct 30 03:00:18.834691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.834672    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"
Oct 30 03:00:18.834826 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.834681    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v" event={"ID":"e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03","Type":"ContainerDied","Data":"61dc84d6cbd0541687bd26bc57269dda0574bd7a6a35fbfc91a030ac3677719a"}
Oct 30 03:00:18.846388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.846183    4278 scope.go:122] "RemoveContainer" containerID="17dff72c8a94aa7649c9d1b2afe13815e65b2d1dfc325c272a686f2a8364612f"
Oct 30 03:00:18.846978 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.846952    4278 generic.go:356] "Generic (PLEG): container finished" podID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerID="a5de5fe915cfb1ef17b505cf1cebf6cacc4e56c0e5e8edb7ab67b7df8044e1ed" exitCode=60
Oct 30 03:00:18.847108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.847084    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" event={"ID":"eda28edd-6a3f-425f-8f62-c6109e64bb5e","Type":"ContainerDied","Data":"a5de5fe915cfb1ef17b505cf1cebf6cacc4e56c0e5e8edb7ab67b7df8044e1ed"}
Oct 30 03:00:18.847698 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.847571    4278 scope.go:122] "RemoveContainer" containerID="a5de5fe915cfb1ef17b505cf1cebf6cacc4e56c0e5e8edb7ab67b7df8044e1ed"
Oct 30 03:00:18.847983 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:18.847939    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm_odh-ai-gateway-infra(eda28edd-6a3f-425f-8f62-c6109e64bb5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e"
Oct 30 03:00:18.849299 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.849198    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"
Oct 30 03:00:18.849299 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.849203    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9" event={"ID":"6ab69754-6a43-4d6e-8e36-ba706d7fe4d9","Type":"ContainerDied","Data":"07a3bdbb78cd89b744b1432d3cd879dd51d01f06731cb1d7b6608f9ca20e60f5"}
Oct 30 03:00:18.857488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.857469    4278 scope.go:122] "RemoveContainer" containerID="c6dd141b6128e8455e7a30eab9d137d889c086384a042ea7592d80eee476db14"
Oct 30 03:00:18.893092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.893062    4278 scope.go:122] "RemoveContainer" containerID="4df0043d7cc9696f709a5a54b1d381b37ca9a71d2a23af1c35b7740783bd3600"
Oct 30 03:00:18.908597 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.908575    4278 scope.go:122] "RemoveContainer" containerID="27969080717cac01489727a54f82dc2c8acd557b445f69e65aa59053d9fee0a8"
Oct 30 03:00:18.917286 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.917256    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 03:00:18.917389 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.917318    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 03:00:18.993474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:18.990727    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"]
Oct 30 03:00:19.010646 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.010617    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"]
Oct 30 03:00:19.016983 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.016950    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888820-rqfl9"]
Oct 30 03:00:19.065458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.058448    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"]
Oct 30 03:00:19.093664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.093573    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e2b0506d-29888820-x7nmr"]
Oct 30 03:00:19.121123 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.121091    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"]
Oct 30 03:00:19.125072 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.125050    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888820-xk29v"]
Oct 30 03:00:19.169313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.168720    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"]
Oct 30 03:00:19.199991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.199929    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888820-pqqr9"]
Oct 30 03:00:19.731442 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.731387    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"]
Oct 30 03:00:19.732215 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732192    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerName="cleanup"
Oct 30 03:00:19.732215 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732212    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732223    4278 cpu_manager.go:414] "containerMap: removing container" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732229    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732239    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732245    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732259    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732264    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732273    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732277    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732286    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732290    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732295    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732299    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732308    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732313    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732324    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732328    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732336    4278 cpu_manager.go:414] "containerMap: removing container" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerName="cleanup"
Oct 30 03:00:19.732369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732341    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732454    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732464    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732471    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732481    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732489    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732496    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732503    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732509    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732513    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732518    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerName="cleanup"
Oct 30 03:00:19.732933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.732523    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" containerName="cleanup"
Oct 30 03:00:19.737481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.737463    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:19.755794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.755768    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"]
Oct 30 03:00:19.789451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.787991    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:00:19.892228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.892191    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0afc2e46-x8v9x\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:19.892739 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.892309    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpz76\" (UniqueName: \"kubernetes.io/projected/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-kube-api-access-bpz76\") pod \"maas-api-revoke-sub-0afc2e46-x8v9x\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:19.892739 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.892595    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0afc2e46-x8v9x\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:19.918252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.918225    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 03:00:19.918385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.918281    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 03:00:19.994504 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.994381    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0afc2e46-x8v9x\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:19.994663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.994530    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpz76\" (UniqueName: \"kubernetes.io/projected/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-kube-api-access-bpz76\") pod \"maas-api-revoke-sub-0afc2e46-x8v9x\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:19.994663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.994644    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0afc2e46-x8v9x\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:19.995587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.995558    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0afc2e46-x8v9x\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:19.998840 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:19.998795    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0afc2e46-x8v9x\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:20.023853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.023815    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpz76\" (UniqueName: \"kubernetes.io/projected/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-kube-api-access-bpz76\") pod \"maas-api-revoke-sub-0afc2e46-x8v9x\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:20.079448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.077677    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:20.142564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.142528    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"
Oct 30 03:00:20.230687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.230655    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" path="/var/lib/kubelet/pods/5e34a96e-72d0-4c46-b988-d12cbfdb8a00/volumes"
Oct 30 03:00:20.231353 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.231335    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2b109e-89e0-4616-bb5f-0be53618f755" path="/var/lib/kubelet/pods/5f2b109e-89e0-4616-bb5f-0be53618f755/volumes"
Oct 30 03:00:20.231925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.231909    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab69754-6a43-4d6e-8e36-ba706d7fe4d9" path="/var/lib/kubelet/pods/6ab69754-6a43-4d6e-8e36-ba706d7fe4d9/volumes"
Oct 30 03:00:20.232730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.232715    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" path="/var/lib/kubelet/pods/e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03/volumes"
Oct 30 03:00:20.310858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.310821    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda28edd-6a3f-425f-8f62-c6109e64bb5e-kube-api-access-4kjml\" (UniqueName: \"kubernetes.io/projected/eda28edd-6a3f-425f-8f62-c6109e64bb5e-kube-api-access-4kjml\") pod \"eda28edd-6a3f-425f-8f62-c6109e64bb5e\" (UID: \"eda28edd-6a3f-425f-8f62-c6109e64bb5e\") "
Oct 30 03:00:20.311131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.311109    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda28edd-6a3f-425f-8f62-c6109e64bb5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eda28edd-6a3f-425f-8f62-c6109e64bb5e-maas-api-service-ca\") pod \"eda28edd-6a3f-425f-8f62-c6109e64bb5e\" (UID: \"eda28edd-6a3f-425f-8f62-c6109e64bb5e\") "
Oct 30 03:00:20.312238 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.312210    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda28edd-6a3f-425f-8f62-c6109e64bb5e-maas-api-service-ca" pod "eda28edd-6a3f-425f-8f62-c6109e64bb5e" (UID: "eda28edd-6a3f-425f-8f62-c6109e64bb5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:20.317643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.314366    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda28edd-6a3f-425f-8f62-c6109e64bb5e-kube-api-access-4kjml" pod "eda28edd-6a3f-425f-8f62-c6109e64bb5e" (UID: "eda28edd-6a3f-425f-8f62-c6109e64bb5e"). InnerVolumeSpecName "kube-api-access-4kjml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:20.413523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.413481    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eda28edd-6a3f-425f-8f62-c6109e64bb5e-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:20.413523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.413515    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kjml\" (UniqueName: \"kubernetes.io/projected/eda28edd-6a3f-425f-8f62-c6109e64bb5e-kube-api-access-4kjml\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:20.432986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.432956    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"]
Oct 30 03:00:20.433117 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:20.432957    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3617e5_ed3c_41fc_bc1e_f164a1b2f937.slice/crio-043918b5ff74bc7cb9ca9abac6284b0cc712955b9ebb1934c05dc3c6201f5a95 WatchSource:0}: Error finding container 043918b5ff74bc7cb9ca9abac6284b0cc712955b9ebb1934c05dc3c6201f5a95: Status 404 returned error can't find the container with id 043918b5ff74bc7cb9ca9abac6284b0cc712955b9ebb1934c05dc3c6201f5a95
Oct 30 03:00:20.499542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.499485    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"]
Oct 30 03:00:20.500312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500287    4278 cpu_manager.go:414] "containerMap: removing container" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerName="cleanup"
Oct 30 03:00:20.500405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500314    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerName="cleanup"
Oct 30 03:00:20.500405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500328    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerName="cleanup"
Oct 30 03:00:20.500405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500335    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerName="cleanup"
Oct 30 03:00:20.500405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500351    4278 cpu_manager.go:414] "containerMap: removing container" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerName="cleanup"
Oct 30 03:00:20.500405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500358    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerName="cleanup"
Oct 30 03:00:20.500607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500535    4278 cpu_manager.go:414] "containerMap: removing container" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerName="cleanup"
Oct 30 03:00:20.500607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500547    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e252e7e1-2bc6-46f4-b532-ab2d5f1ecf03" containerName="cleanup"
Oct 30 03:00:20.500673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500669    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerName="cleanup"
Oct 30 03:00:20.500706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500681    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerName="cleanup"
Oct 30 03:00:20.500706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500687    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e34a96e-72d0-4c46-b988-d12cbfdb8a00" containerName="cleanup"
Oct 30 03:00:20.500706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.500692    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerName="cleanup"
Oct 30 03:00:20.510888 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.510866    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"]
Oct 30 03:00:20.511001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.510952    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:20.616640 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.616544    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jctmn\" (UniqueName: \"kubernetes.io/projected/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-kube-api-access-jctmn\") pod \"maas-api-revoke-sub-c7783122-jxgcg\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:20.616809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.616714    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7783122-jxgcg\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:20.617011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.616991    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7783122-jxgcg\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:20.732384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.720229    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7783122-jxgcg\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:20.732384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.720314    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jctmn\" (UniqueName: \"kubernetes.io/projected/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-kube-api-access-jctmn\") pod \"maas-api-revoke-sub-c7783122-jxgcg\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:20.732384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.720419    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7783122-jxgcg\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:20.732384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.721471    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7783122-jxgcg\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:20.769453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.768715    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7783122-jxgcg\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:20.868524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.862553    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jctmn\" (UniqueName: \"kubernetes.io/projected/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-kube-api-access-jctmn\") pod \"maas-api-revoke-sub-c7783122-jxgcg\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:20.909379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.908618    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x" event={"ID":"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937","Type":"ContainerStarted","Data":"8384c36aaf6a221dbaaac4a789d7bd38113846e0af5ed7cbb9291b954f33b3ac"}
Oct 30 03:00:20.909862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.909800    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x" event={"ID":"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937","Type":"ContainerStarted","Data":"043918b5ff74bc7cb9ca9abac6284b0cc712955b9ebb1934c05dc3c6201f5a95"}
Oct 30 03:00:20.911592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.910787    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"
Oct 30 03:00:20.911592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.910816    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm" event={"ID":"eda28edd-6a3f-425f-8f62-c6109e64bb5e","Type":"ContainerDied","Data":"cd0f4e2cb30edc7303c4a08f68d020a10b3927ea1b45f807232502577c685ddc"}
Oct 30 03:00:20.911592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.910864    4278 scope.go:122] "RemoveContainer" containerID="a5de5fe915cfb1ef17b505cf1cebf6cacc4e56c0e5e8edb7ab67b7df8044e1ed"
Oct 30 03:00:20.917711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.917599    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 03:00:20.917711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.917656    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 03:00:20.952373 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:20.952324    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x" podStartSLOduration=1.9523111549999999 podStartE2EDuration="1.952311155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:20.949444798 +0000 UTC m=+1158.873652992" watchObservedRunningTime="2026-10-30 03:00:20.952311155 +0000 UTC m=+1158.876519343"
Oct 30 03:00:21.011762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.011727    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"]
Oct 30 03:00:21.018978 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.018952    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888820-ghwgm"]
Oct 30 03:00:21.129110 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.129027    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:21.499752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.496827    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"]
Oct 30 03:00:21.500690 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:21.500663    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c7e2c7_06ab_4253_ac03_e6d21c655dd3.slice/crio-c09f171e4725f9836649105529e20b061140c73bb8ab1e1d0180ebd905254f3e WatchSource:0}: Error finding container c09f171e4725f9836649105529e20b061140c73bb8ab1e1d0180ebd905254f3e: Status 404 returned error can't find the container with id c09f171e4725f9836649105529e20b061140c73bb8ab1e1d0180ebd905254f3e
Oct 30 03:00:21.917391 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.917358    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 03:00:21.917810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.917437    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 03:00:21.918974 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.918944    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg" event={"ID":"94c7e2c7-06ab-4253-ac03-e6d21c655dd3","Type":"ContainerStarted","Data":"3a6b516a5457a4acf19f49b15e53a309fdecac4d8a3bbc3ef6744b8f7ddf6d69"}
Oct 30 03:00:21.919097 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.919002    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg" event={"ID":"94c7e2c7-06ab-4253-ac03-e6d21c655dd3","Type":"ContainerStarted","Data":"c09f171e4725f9836649105529e20b061140c73bb8ab1e1d0180ebd905254f3e"}
Oct 30 03:00:21.920608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.920585    4278 generic.go:356] "Generic (PLEG): container finished" podID="ab3617e5-ed3c-41fc-bc1e-f164a1b2f937" containerID="8384c36aaf6a221dbaaac4a789d7bd38113846e0af5ed7cbb9291b954f33b3ac" exitCode=0
Oct 30 03:00:21.920693 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.920662    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x" event={"ID":"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937","Type":"ContainerDied","Data":"8384c36aaf6a221dbaaac4a789d7bd38113846e0af5ed7cbb9291b954f33b3ac"}
Oct 30 03:00:21.972707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:21.972641    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg" podStartSLOduration=1.972626357 podStartE2EDuration="1.972626357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:21.970608301 +0000 UTC m=+1159.894816501" watchObservedRunningTime="2026-10-30 03:00:21.972626357 +0000 UTC m=+1159.896834545"
Oct 30 03:00:22.202702 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.202603    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" path="/var/lib/kubelet/pods/eda28edd-6a3f-425f-8f62-c6109e64bb5e/volumes"
Oct 30 03:00:22.712058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.711980    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"]
Oct 30 03:00:22.712692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.712676    4278 cpu_manager.go:414] "containerMap: removing container" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerName="cleanup"
Oct 30 03:00:22.712692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.712692    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda28edd-6a3f-425f-8f62-c6109e64bb5e" containerName="cleanup"
Oct 30 03:00:22.716275 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.716259    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:22.739383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.739353    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"]
Oct 30 03:00:22.858712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.858674    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:22.858872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.858858    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:22.858968 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.858948    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xql8b\" (UniqueName: \"kubernetes.io/projected/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-kube-api-access-xql8b\") pod \"maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:22.931629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.931583    4278 generic.go:356] "Generic (PLEG): container finished" podID="94c7e2c7-06ab-4253-ac03-e6d21c655dd3" containerID="3a6b516a5457a4acf19f49b15e53a309fdecac4d8a3bbc3ef6744b8f7ddf6d69" exitCode=0
Oct 30 03:00:22.932088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.931664    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg" event={"ID":"94c7e2c7-06ab-4253-ac03-e6d21c655dd3","Type":"ContainerDied","Data":"3a6b516a5457a4acf19f49b15e53a309fdecac4d8a3bbc3ef6744b8f7ddf6d69"}
Oct 30 03:00:22.961002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.960964    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:22.961181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.961102    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:22.961181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.961146    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xql8b\" (UniqueName: \"kubernetes.io/projected/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-kube-api-access-xql8b\") pod \"maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:22.962006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.961974    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:22.970501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.963830    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:22.982640 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:22.982399    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xql8b\" (UniqueName: \"kubernetes.io/projected/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-kube-api-access-xql8b\") pod \"maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:23.030747 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.030713    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:23.230125 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.230038    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:23.380731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.380702    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-openshift-ca\") pod \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") "
Oct 30 03:00:23.380862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.380775    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-kube-api-access-bpz76\" (UniqueName: \"kubernetes.io/projected/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-kube-api-access-bpz76\") pod \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") "
Oct 30 03:00:23.380924 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.380886    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-xks-ca\") pod \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\" (UID: \"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937\") "
Oct 30 03:00:23.381148 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.381109    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-openshift-ca" pod "ab3617e5-ed3c-41fc-bc1e-f164a1b2f937" (UID: "ab3617e5-ed3c-41fc-bc1e-f164a1b2f937"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:23.381769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.381742    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:23.382284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.382255    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"]
Oct 30 03:00:23.383098 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:23.382750    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79530fd5_67b0_45b1_ba8a_8c271cd44fd6.slice/crio-3350abf6a3d16a9293906777908cac13064e740d7282184b535b4e7ff9222e92 WatchSource:0}: Error finding container 3350abf6a3d16a9293906777908cac13064e740d7282184b535b4e7ff9222e92: Status 404 returned error can't find the container with id 3350abf6a3d16a9293906777908cac13064e740d7282184b535b4e7ff9222e92
Oct 30 03:00:23.383214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.383194    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-xks-ca" pod "ab3617e5-ed3c-41fc-bc1e-f164a1b2f937" (UID: "ab3617e5-ed3c-41fc-bc1e-f164a1b2f937"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:23.383294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.383280    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-kube-api-access-bpz76" pod "ab3617e5-ed3c-41fc-bc1e-f164a1b2f937" (UID: "ab3617e5-ed3c-41fc-bc1e-f164a1b2f937"). InnerVolumeSpecName "kube-api-access-bpz76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:23.484142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.484053    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpz76\" (UniqueName: \"kubernetes.io/projected/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-kube-api-access-bpz76\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:23.484142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.484094    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:23.512502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.512466    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:23.513315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.513283    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:23.514165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.514142    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:00:23.940774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.940735    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d" event={"ID":"79530fd5-67b0-45b1-ba8a-8c271cd44fd6","Type":"ContainerStarted","Data":"4babcf2b5ba5fb84e60a257d46738b72b33ac70239464e0ae29a9a4b54f72540"}
Oct 30 03:00:23.941219 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.940795    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d" event={"ID":"79530fd5-67b0-45b1-ba8a-8c271cd44fd6","Type":"ContainerStarted","Data":"3350abf6a3d16a9293906777908cac13064e740d7282184b535b4e7ff9222e92"}
Oct 30 03:00:23.942272 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.942249    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"
Oct 30 03:00:23.942436 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.942273    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x" event={"ID":"ab3617e5-ed3c-41fc-bc1e-f164a1b2f937","Type":"ContainerDied","Data":"043918b5ff74bc7cb9ca9abac6284b0cc712955b9ebb1934c05dc3c6201f5a95"}
Oct 30 03:00:23.942436 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.942323    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="043918b5ff74bc7cb9ca9abac6284b0cc712955b9ebb1934c05dc3c6201f5a95"
Oct 30 03:00:23.985797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:23.985743    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d" podStartSLOduration=1.985726909 podStartE2EDuration="1.985726909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:23.981957453 +0000 UTC m=+1161.906165644" watchObservedRunningTime="2026-10-30 03:00:23.985726909 +0000 UTC m=+1161.909935096"
Oct 30 03:00:24.121917 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.121894    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:24.194759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.194719    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-openshift-ca\") pod \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") "
Oct 30 03:00:24.194962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.194832    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-kube-api-access-jctmn\" (UniqueName: \"kubernetes.io/projected/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-kube-api-access-jctmn\") pod \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") "
Oct 30 03:00:24.194962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.194935    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-xks-ca\") pod \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\" (UID: \"94c7e2c7-06ab-4253-ac03-e6d21c655dd3\") "
Oct 30 03:00:24.196129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.195898    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-openshift-ca" pod "94c7e2c7-06ab-4253-ac03-e6d21c655dd3" (UID: "94c7e2c7-06ab-4253-ac03-e6d21c655dd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:24.197574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.197024    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:24.201912 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.201743    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-xks-ca" pod "94c7e2c7-06ab-4253-ac03-e6d21c655dd3" (UID: "94c7e2c7-06ab-4253-ac03-e6d21c655dd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:24.201912 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.201884    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-kube-api-access-jctmn" pod "94c7e2c7-06ab-4253-ac03-e6d21c655dd3" (UID: "94c7e2c7-06ab-4253-ac03-e6d21c655dd3"). InnerVolumeSpecName "kube-api-access-jctmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:24.216801 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:24.216767    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:00:24.298659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.298631    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jctmn\" (UniqueName: \"kubernetes.io/projected/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-kube-api-access-jctmn\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:24.298659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.298655    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7e2c7-06ab-4253-ac03-e6d21c655dd3-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:24.953444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.952469    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"
Oct 30 03:00:24.953444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.952481    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg" event={"ID":"94c7e2c7-06ab-4253-ac03-e6d21c655dd3","Type":"ContainerDied","Data":"c09f171e4725f9836649105529e20b061140c73bb8ab1e1d0180ebd905254f3e"}
Oct 30 03:00:24.953444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.952538    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c09f171e4725f9836649105529e20b061140c73bb8ab1e1d0180ebd905254f3e"
Oct 30 03:00:24.954515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.954487    4278 generic.go:356] "Generic (PLEG): container finished" podID="79530fd5-67b0-45b1-ba8a-8c271cd44fd6" containerID="4babcf2b5ba5fb84e60a257d46738b72b33ac70239464e0ae29a9a4b54f72540" exitCode=0
Oct 30 03:00:24.954675 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:24.954583    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d" event={"ID":"79530fd5-67b0-45b1-ba8a-8c271cd44fd6","Type":"ContainerDied","Data":"4babcf2b5ba5fb84e60a257d46738b72b33ac70239464e0ae29a9a4b54f72540"}
Oct 30 03:00:26.166145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.166114    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:26.293054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.293018    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-openshift-ca\") pod \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") "
Oct 30 03:00:26.293232 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.293071    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-kube-api-access-xql8b\" (UniqueName: \"kubernetes.io/projected/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-kube-api-access-xql8b\") pod \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") "
Oct 30 03:00:26.293342 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.293323    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-xks-ca\") pod \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\" (UID: \"79530fd5-67b0-45b1-ba8a-8c271cd44fd6\") "
Oct 30 03:00:26.293490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.293460    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-openshift-ca" pod "79530fd5-67b0-45b1-ba8a-8c271cd44fd6" (UID: "79530fd5-67b0-45b1-ba8a-8c271cd44fd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:26.294498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.294332    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:26.296342 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.295983    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-kube-api-access-xql8b" pod "79530fd5-67b0-45b1-ba8a-8c271cd44fd6" (UID: "79530fd5-67b0-45b1-ba8a-8c271cd44fd6"). InnerVolumeSpecName "kube-api-access-xql8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:26.296342 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.297307    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-xks-ca" pod "79530fd5-67b0-45b1-ba8a-8c271cd44fd6" (UID: "79530fd5-67b0-45b1-ba8a-8c271cd44fd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:26.395595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.395556    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:26.395595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.395588    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xql8b\" (UniqueName: \"kubernetes.io/projected/79530fd5-67b0-45b1-ba8a-8c271cd44fd6-kube-api-access-xql8b\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:26.972616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.972573    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"
Oct 30 03:00:26.972840 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.972576    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d" event={"ID":"79530fd5-67b0-45b1-ba8a-8c271cd44fd6","Type":"ContainerDied","Data":"3350abf6a3d16a9293906777908cac13064e740d7282184b535b4e7ff9222e92"}
Oct 30 03:00:26.972840 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:26.972727    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3350abf6a3d16a9293906777908cac13064e740d7282184b535b4e7ff9222e92"
Oct 30 03:00:27.274915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.274884    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"]
Oct 30 03:00:27.275593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.275572    4278 cpu_manager.go:414] "containerMap: removing container" podUID="79530fd5-67b0-45b1-ba8a-8c271cd44fd6" containerName="revoke-keys"
Oct 30 03:00:27.275593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.275591    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79530fd5-67b0-45b1-ba8a-8c271cd44fd6" containerName="revoke-keys"
Oct 30 03:00:27.275779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.275608    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ab3617e5-ed3c-41fc-bc1e-f164a1b2f937" containerName="revoke-keys"
Oct 30 03:00:27.275779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.275613    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3617e5-ed3c-41fc-bc1e-f164a1b2f937" containerName="revoke-keys"
Oct 30 03:00:27.275779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.275628    4278 cpu_manager.go:414] "containerMap: removing container" podUID="94c7e2c7-06ab-4253-ac03-e6d21c655dd3" containerName="revoke-keys"
Oct 30 03:00:27.275779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.275634    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c7e2c7-06ab-4253-ac03-e6d21c655dd3" containerName="revoke-keys"
Oct 30 03:00:27.275779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.275752    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="79530fd5-67b0-45b1-ba8a-8c271cd44fd6" containerName="revoke-keys"
Oct 30 03:00:27.275779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.275762    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3617e5-ed3c-41fc-bc1e-f164a1b2f937" containerName="revoke-keys"
Oct 30 03:00:27.275779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.275769    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c7e2c7-06ab-4253-ac03-e6d21c655dd3" containerName="revoke-keys"
Oct 30 03:00:27.279729 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.279709    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:27.298729 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.298688    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"]
Oct 30 03:00:27.306605 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.305519    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:00:27.306605 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.305801    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:00:27.411142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.411103    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4b5s\" (UniqueName: \"kubernetes.io/projected/9a0843f8-3f48-4920-a716-3043351d1ea4-kube-api-access-p4b5s\") pod \"maas-api-revoke-sub-b1e6d2f3-zz6wf\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:27.411337 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.411209    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1e6d2f3-zz6wf\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:27.411402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.411378    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1e6d2f3-zz6wf\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:27.514446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.513802    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1e6d2f3-zz6wf\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:27.514446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.513969    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4b5s\" (UniqueName: \"kubernetes.io/projected/9a0843f8-3f48-4920-a716-3043351d1ea4-kube-api-access-p4b5s\") pod \"maas-api-revoke-sub-b1e6d2f3-zz6wf\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:27.514446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.514039    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1e6d2f3-zz6wf\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:27.519495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.515388    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1e6d2f3-zz6wf\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:27.523508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.520151    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1e6d2f3-zz6wf\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:27.537029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.536950    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4b5s\" (UniqueName: \"kubernetes.io/projected/9a0843f8-3f48-4920-a716-3043351d1ea4-kube-api-access-p4b5s\") pod \"maas-api-revoke-sub-b1e6d2f3-zz6wf\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:27.601626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:27.601584    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:28.131448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.130483    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"]
Oct 30 03:00:28.215479 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.192331    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"]
Oct 30 03:00:28.306676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.306509    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9198fe5f-8npqn"]
Oct 30 03:00:28.307193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.307173    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"]
Oct 30 03:00:28.627023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.626984    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf" event={"ID":"9a0843f8-3f48-4920-a716-3043351d1ea4","Type":"ContainerStarted","Data":"ab07d60b46fed792a2538636d998321715e3c9a6c76ba396355e8d9e44293899"}
Oct 30 03:00:28.671585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.671549    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"]
Oct 30 03:00:28.671776 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.653130    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:28.688379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.688321    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4b5g\" (UniqueName: \"kubernetes.io/projected/253b29df-ffcc-4d70-9c91-4954b5c28be7-kube-api-access-x4b5g\") pod \"maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:28.688755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.688732    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:28.688905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.688887    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:28.792807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.791530    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4b5g\" (UniqueName: \"kubernetes.io/projected/253b29df-ffcc-4d70-9c91-4954b5c28be7-kube-api-access-x4b5g\") pod \"maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:28.792807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.791659    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:28.792807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.791690    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:28.797454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.794554    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:28.890508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.887317    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:28.916535 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:28.916491    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4b5g\" (UniqueName: \"kubernetes.io/projected/253b29df-ffcc-4d70-9c91-4954b5c28be7-kube-api-access-x4b5g\") pod \"maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:29.107450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.104295    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:29.138453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.136670    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:29.149508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.148751    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:29.164588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.164532    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:29.190876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.190824    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:29.312337 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.311759    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:29.323871 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.323582    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:29.449512 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.448759    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:29.450778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.449755    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:29.673619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.673574    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:29.731055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:29.730902    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:30.123448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.111518    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:30.123448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.112304    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 03:00:30.151008 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.150968    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:30.152228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.152204    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 03:00:30.216646 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.165024    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 30 03:00:30.433065 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:30.433029    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253b29df_ffcc_4d70_9c91_4954b5c28be7.slice/crio-62951cea553779e56dc71a5eb4ab916ed03afb3c6bca7ecef88bf4cc70dae80e WatchSource:0}: Error finding container 62951cea553779e56dc71a5eb4ab916ed03afb3c6bca7ecef88bf4cc70dae80e: Status 404 returned error can't find the container with id 62951cea553779e56dc71a5eb4ab916ed03afb3c6bca7ecef88bf4cc70dae80e
Oct 30 03:00:30.449465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.447653    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815a4d4b-13ca-4a7a-a4b2-75c32c402881" path="/var/lib/kubelet/pods/815a4d4b-13ca-4a7a-a4b2-75c32c402881/volumes"
Oct 30 03:00:30.449465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.448148    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 03:00:30.466932 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.466903    4278 generic.go:356] "Generic (PLEG): container finished" podID="9a0843f8-3f48-4920-a716-3043351d1ea4" containerID="bfdfecbcad6ce5322220e292c4d983715bcc38290f13d1815e848eb9d4a785bc" exitCode=0
Oct 30 03:00:30.467273 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.467249    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf" event={"ID":"9a0843f8-3f48-4920-a716-3043351d1ea4","Type":"ContainerDied","Data":"bfdfecbcad6ce5322220e292c4d983715bcc38290f13d1815e848eb9d4a785bc"}
Oct 30 03:00:30.478543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.478466    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb" event={"ID":"253b29df-ffcc-4d70-9c91-4954b5c28be7","Type":"ContainerStarted","Data":"62951cea553779e56dc71a5eb4ab916ed03afb3c6bca7ecef88bf4cc70dae80e"}
Oct 30 03:00:30.538853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.538826    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 03:00:30.556025 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.542840    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 03:00:30.563536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.563510    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 03:00:30.654655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:30.654573    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"]
Oct 30 03:00:31.103449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.090539    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"]
Oct 30 03:00:31.115462 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.115051    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6299da6-llhc8"]
Oct 30 03:00:31.521848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.521815    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb" event={"ID":"253b29df-ffcc-4d70-9c91-4954b5c28be7","Type":"ContainerStarted","Data":"95e7079d381ae5e41d5a28efe6d01c9ac24843ae66b45688da071c6d2ecfbc8d"}
Oct 30 03:00:31.598447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.593657    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb" podStartSLOduration=3.593636848 podStartE2EDuration="3.593636848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:31.567877586 +0000 UTC m=+1169.492085788" watchObservedRunningTime="2026-10-30 03:00:31.593636848 +0000 UTC m=+1169.517845037"
Oct 30 03:00:31.793544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.792228    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:31.903303 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.903264    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-openshift-ca\") pod \"9a0843f8-3f48-4920-a716-3043351d1ea4\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") "
Oct 30 03:00:31.903618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.903595    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-xks-ca\") pod \"9a0843f8-3f48-4920-a716-3043351d1ea4\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") "
Oct 30 03:00:31.903714 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.903692    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0843f8-3f48-4920-a716-3043351d1ea4-kube-api-access-p4b5s\" (UniqueName: \"kubernetes.io/projected/9a0843f8-3f48-4920-a716-3043351d1ea4-kube-api-access-p4b5s\") pod \"9a0843f8-3f48-4920-a716-3043351d1ea4\" (UID: \"9a0843f8-3f48-4920-a716-3043351d1ea4\") "
Oct 30 03:00:31.903791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.903766    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-openshift-ca" pod "9a0843f8-3f48-4920-a716-3043351d1ea4" (UID: "9a0843f8-3f48-4920-a716-3043351d1ea4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:31.904552 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.904524    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:31.908235 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.908207    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-xks-ca" pod "9a0843f8-3f48-4920-a716-3043351d1ea4" (UID: "9a0843f8-3f48-4920-a716-3043351d1ea4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:31.911664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:31.911639    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0843f8-3f48-4920-a716-3043351d1ea4-kube-api-access-p4b5s" pod "9a0843f8-3f48-4920-a716-3043351d1ea4" (UID: "9a0843f8-3f48-4920-a716-3043351d1ea4"). InnerVolumeSpecName "kube-api-access-p4b5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:32.006503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:32.006417    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0843f8-3f48-4920-a716-3043351d1ea4-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:32.006503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:32.006494    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4b5s\" (UniqueName: \"kubernetes.io/projected/9a0843f8-3f48-4920-a716-3043351d1ea4-kube-api-access-p4b5s\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:32.213827 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:32.213747    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a788c64-036f-47b7-a8ea-e415b0ee2f13" path="/var/lib/kubelet/pods/6a788c64-036f-47b7-a8ea-e415b0ee2f13/volumes"
Oct 30 03:00:32.530896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:32.530858    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"
Oct 30 03:00:32.531488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:32.530892    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf" event={"ID":"9a0843f8-3f48-4920-a716-3043351d1ea4","Type":"ContainerDied","Data":"ab07d60b46fed792a2538636d998321715e3c9a6c76ba396355e8d9e44293899"}
Oct 30 03:00:32.531488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:32.530940    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab07d60b46fed792a2538636d998321715e3c9a6c76ba396355e8d9e44293899"
Oct 30 03:00:32.532741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:32.532712    4278 generic.go:356] "Generic (PLEG): container finished" podID="253b29df-ffcc-4d70-9c91-4954b5c28be7" containerID="95e7079d381ae5e41d5a28efe6d01c9ac24843ae66b45688da071c6d2ecfbc8d" exitCode=0
Oct 30 03:00:32.532859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:32.532756    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb" event={"ID":"253b29df-ffcc-4d70-9c91-4954b5c28be7","Type":"ContainerDied","Data":"95e7079d381ae5e41d5a28efe6d01c9ac24843ae66b45688da071c6d2ecfbc8d"}
Oct 30 03:00:33.517033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:33.516998    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"]
Oct 30 03:00:33.551562 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:33.551530    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"]
Oct 30 03:00:33.835163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:33.835138    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:33.932976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:33.932944    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-xks-ca\") pod \"253b29df-ffcc-4d70-9c91-4954b5c28be7\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") "
Oct 30 03:00:33.933169 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:33.933122    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-openshift-ca\") pod \"253b29df-ffcc-4d70-9c91-4954b5c28be7\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") "
Oct 30 03:00:33.933169 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:33.933153    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253b29df-ffcc-4d70-9c91-4954b5c28be7-kube-api-access-x4b5g\" (UniqueName: \"kubernetes.io/projected/253b29df-ffcc-4d70-9c91-4954b5c28be7-kube-api-access-x4b5g\") pod \"253b29df-ffcc-4d70-9c91-4954b5c28be7\" (UID: \"253b29df-ffcc-4d70-9c91-4954b5c28be7\") "
Oct 30 03:00:33.934133 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:33.934106    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-openshift-ca" pod "253b29df-ffcc-4d70-9c91-4954b5c28be7" (UID: "253b29df-ffcc-4d70-9c91-4954b5c28be7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:33.936103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:33.936056    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-xks-ca" pod "253b29df-ffcc-4d70-9c91-4954b5c28be7" (UID: "253b29df-ffcc-4d70-9c91-4954b5c28be7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:33.936612 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:33.936586    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253b29df-ffcc-4d70-9c91-4954b5c28be7-kube-api-access-x4b5g" pod "253b29df-ffcc-4d70-9c91-4954b5c28be7" (UID: "253b29df-ffcc-4d70-9c91-4954b5c28be7"). InnerVolumeSpecName "kube-api-access-x4b5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:34.034681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:34.034644    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:34.034681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:34.034681    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253b29df-ffcc-4d70-9c91-4954b5c28be7-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:34.034893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:34.034691    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4b5g\" (UniqueName: \"kubernetes.io/projected/253b29df-ffcc-4d70-9c91-4954b5c28be7-kube-api-access-x4b5g\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:34.555368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:34.555341    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"
Oct 30 03:00:34.555877 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:34.555376    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb" event={"ID":"253b29df-ffcc-4d70-9c91-4954b5c28be7","Type":"ContainerDied","Data":"62951cea553779e56dc71a5eb4ab916ed03afb3c6bca7ecef88bf4cc70dae80e"}
Oct 30 03:00:34.555877 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:34.555474    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62951cea553779e56dc71a5eb4ab916ed03afb3c6bca7ecef88bf4cc70dae80e"
Oct 30 03:00:37.559482 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.559449    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"]
Oct 30 03:00:37.560189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.560170    4278 cpu_manager.go:414] "containerMap: removing container" podUID="9a0843f8-3f48-4920-a716-3043351d1ea4" containerName="revoke-keys"
Oct 30 03:00:37.560244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.560191    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0843f8-3f48-4920-a716-3043351d1ea4" containerName="revoke-keys"
Oct 30 03:00:37.560244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.560222    4278 cpu_manager.go:414] "containerMap: removing container" podUID="253b29df-ffcc-4d70-9c91-4954b5c28be7" containerName="revoke-keys"
Oct 30 03:00:37.560244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.560227    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253b29df-ffcc-4d70-9c91-4954b5c28be7" containerName="revoke-keys"
Oct 30 03:00:37.560352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.560343    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0843f8-3f48-4920-a716-3043351d1ea4" containerName="revoke-keys"
Oct 30 03:00:37.560386 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.560355    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="253b29df-ffcc-4d70-9c91-4954b5c28be7" containerName="revoke-keys"
Oct 30 03:00:37.564544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.564525    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:37.580539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.580500    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:00:37.586410 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.586384    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"]
Oct 30 03:00:37.593278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.593246    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9305c8f8-fj5vr\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:37.593566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.593546    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9305c8f8-fj5vr\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:37.593610 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.593597    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69rnl\" (UniqueName: \"kubernetes.io/projected/656f02c8-4d75-42e4-8600-0861f87e2828-kube-api-access-69rnl\") pod \"maas-api-revoke-sub-9305c8f8-fj5vr\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:37.607705 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.607674    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:00:37.669461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.669399    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"]
Oct 30 03:00:37.674281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.674259    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:37.699312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.699264    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"]
Oct 30 03:00:37.710038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.709997    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tjm8\" (UniqueName: \"kubernetes.io/projected/1207dcbb-33f8-4776-9ea5-01f3f45ba596-kube-api-access-5tjm8\") pod \"maas-api-revoke-keys-e2e-ait-adca906b-5fx7l\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:37.710221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.710063    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9305c8f8-fj5vr\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:37.710221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.710111    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69rnl\" (UniqueName: \"kubernetes.io/projected/656f02c8-4d75-42e4-8600-0861f87e2828-kube-api-access-69rnl\") pod \"maas-api-revoke-sub-9305c8f8-fj5vr\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:37.710221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.710173    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9305c8f8-fj5vr\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:37.710221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.710204    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-adca906b-5fx7l\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:37.710511 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.710276    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-adca906b-5fx7l\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:37.711210 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.711182    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9305c8f8-fj5vr\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:37.713943 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.713918    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9305c8f8-fj5vr\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:37.730356 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.730333    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69rnl\" (UniqueName: \"kubernetes.io/projected/656f02c8-4d75-42e4-8600-0861f87e2828-kube-api-access-69rnl\") pod \"maas-api-revoke-sub-9305c8f8-fj5vr\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:37.812004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.811928    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tjm8\" (UniqueName: \"kubernetes.io/projected/1207dcbb-33f8-4776-9ea5-01f3f45ba596-kube-api-access-5tjm8\") pod \"maas-api-revoke-keys-e2e-ait-adca906b-5fx7l\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:37.812150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.812043    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-adca906b-5fx7l\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:37.812150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.812105    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-adca906b-5fx7l\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:37.812749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.812715    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-adca906b-5fx7l\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:37.814764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.814739    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-adca906b-5fx7l\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:37.832328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.832300    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tjm8\" (UniqueName: \"kubernetes.io/projected/1207dcbb-33f8-4776-9ea5-01f3f45ba596-kube-api-access-5tjm8\") pod \"maas-api-revoke-keys-e2e-ait-adca906b-5fx7l\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:37.880391 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:37.880357    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:38.002189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.001575    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:38.191259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.191230    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"]
Oct 30 03:00:38.192153 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:38.192119    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656f02c8_4d75_42e4_8600_0861f87e2828.slice/crio-48857db74af0d3b6b13e831a22483d4f153c1886af7bd4829792144ad192c809 WatchSource:0}: Error finding container 48857db74af0d3b6b13e831a22483d4f153c1886af7bd4829792144ad192c809: Status 404 returned error can't find the container with id 48857db74af0d3b6b13e831a22483d4f153c1886af7bd4829792144ad192c809
Oct 30 03:00:38.311221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.311190    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"]
Oct 30 03:00:38.311900 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:38.311873    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1207dcbb_33f8_4776_9ea5_01f3f45ba596.slice/crio-32ba6e2262ee270d3a33f013e1f6dac1bf6f06f9a48da26f18b651c22197b1a7 WatchSource:0}: Error finding container 32ba6e2262ee270d3a33f013e1f6dac1bf6f06f9a48da26f18b651c22197b1a7: Status 404 returned error can't find the container with id 32ba6e2262ee270d3a33f013e1f6dac1bf6f06f9a48da26f18b651c22197b1a7
Oct 30 03:00:38.517649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.517595    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk" podUID="75485640-6686-42ef-b6fe-c9834e405540" containerName="payload-processing" containerID="cri-o://36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a" gracePeriod=25
Oct 30 03:00:38.551872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.551814    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw" podUID="da5c5135-1f96-4fa4-b97b-b8fcd82eb187" containerName="payload-pre-processing" containerID="cri-o://46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd" gracePeriod=25
Oct 30 03:00:38.619177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.619147    4278 generic.go:356] "Generic (PLEG): container finished" podID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerID="f9a4436cc155887b6d8b4dd94768cb3c0e9dfc121e7247e9dd37a4c03c7e2f03" exitCode=0
Oct 30 03:00:38.619577 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.619215    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" event={"ID":"89c464f3-8c2e-4996-b4c3-7361c529384d","Type":"ContainerDied","Data":"f9a4436cc155887b6d8b4dd94768cb3c0e9dfc121e7247e9dd37a4c03c7e2f03"}
Oct 30 03:00:38.619779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.619759    4278 scope.go:122] "RemoveContainer" containerID="f9a4436cc155887b6d8b4dd94768cb3c0e9dfc121e7247e9dd37a4c03c7e2f03"
Oct 30 03:00:38.620866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.620840    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l" event={"ID":"1207dcbb-33f8-4776-9ea5-01f3f45ba596","Type":"ContainerStarted","Data":"32ba6e2262ee270d3a33f013e1f6dac1bf6f06f9a48da26f18b651c22197b1a7"}
Oct 30 03:00:38.621851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.621819    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:38.621956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.621895    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:38.621956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.621949    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:38.623196 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.623173    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr" event={"ID":"656f02c8-4d75-42e4-8600-0861f87e2828","Type":"ContainerStarted","Data":"ab456413fd34d7422604d5aef3a0b5e4bbadf60bd590e763729039bdef04f75e"}
Oct 30 03:00:38.623298 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.623220    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr" event={"ID":"656f02c8-4d75-42e4-8600-0861f87e2828","Type":"ContainerStarted","Data":"48857db74af0d3b6b13e831a22483d4f153c1886af7bd4829792144ad192c809"}
Oct 30 03:00:38.692816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.692362    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"]
Oct 30 03:00:38.715099 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.714467    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr" podStartSLOduration=1.714447246 podStartE2EDuration="1.714447246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:38.706717563 +0000 UTC m=+1176.630925755" watchObservedRunningTime="2026-10-30 03:00:38.714447246 +0000 UTC m=+1176.638655434"
Oct 30 03:00:38.923879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.922299    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 03:00:38.984671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:38.983837    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 03:00:39.038658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.038595    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-plugins-config-volume\") pod \"da5c5135-1f96-4fa4-b97b-b8fcd82eb187\" (UID: \"da5c5135-1f96-4fa4-b97b-b8fcd82eb187\") "
Oct 30 03:00:39.039242 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.039214    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75485640-6686-42ef-b6fe-c9834e405540-kube-api-access-c9bv5\" (UniqueName: \"kubernetes.io/projected/75485640-6686-42ef-b6fe-c9834e405540-kube-api-access-c9bv5\") pod \"75485640-6686-42ef-b6fe-c9834e405540\" (UID: \"75485640-6686-42ef-b6fe-c9834e405540\") "
Oct 30 03:00:39.039730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.039692    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75485640-6686-42ef-b6fe-c9834e405540-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75485640-6686-42ef-b6fe-c9834e405540-plugins-config-volume\") pod \"75485640-6686-42ef-b6fe-c9834e405540\" (UID: \"75485640-6686-42ef-b6fe-c9834e405540\") "
Oct 30 03:00:39.039914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.039893    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-kube-api-access-rh7xq\" (UniqueName: \"kubernetes.io/projected/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-kube-api-access-rh7xq\") pod \"da5c5135-1f96-4fa4-b97b-b8fcd82eb187\" (UID: \"da5c5135-1f96-4fa4-b97b-b8fcd82eb187\") "
Oct 30 03:00:39.042034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.041991    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75485640-6686-42ef-b6fe-c9834e405540-plugins-config-volume" pod "75485640-6686-42ef-b6fe-c9834e405540" (UID: "75485640-6686-42ef-b6fe-c9834e405540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:39.042525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.042499    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-plugins-config-volume" pod "da5c5135-1f96-4fa4-b97b-b8fcd82eb187" (UID: "da5c5135-1f96-4fa4-b97b-b8fcd82eb187"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:39.043688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.043657    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-kube-api-access-rh7xq" pod "da5c5135-1f96-4fa4-b97b-b8fcd82eb187" (UID: "da5c5135-1f96-4fa4-b97b-b8fcd82eb187"). InnerVolumeSpecName "kube-api-access-rh7xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:39.044465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.044394    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75485640-6686-42ef-b6fe-c9834e405540-kube-api-access-c9bv5" pod "75485640-6686-42ef-b6fe-c9834e405540" (UID: "75485640-6686-42ef-b6fe-c9834e405540"). InnerVolumeSpecName "kube-api-access-c9bv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:39.142642 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.142547    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75485640-6686-42ef-b6fe-c9834e405540-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:39.142642 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.142581    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh7xq\" (UniqueName: \"kubernetes.io/projected/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-kube-api-access-rh7xq\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:39.142642 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.142594    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da5c5135-1f96-4fa4-b97b-b8fcd82eb187-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:39.142642 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.142619    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bv5\" (UniqueName: \"kubernetes.io/projected/75485640-6686-42ef-b6fe-c9834e405540-kube-api-access-c9bv5\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:39.548918 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.548883    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 03:00:39.642277 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.642243    4278 generic.go:356] "Generic (PLEG): container finished" podID="75485640-6686-42ef-b6fe-c9834e405540" containerID="36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a" exitCode=0
Oct 30 03:00:39.642744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.642317    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"
Oct 30 03:00:39.642744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.642330    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk" event={"ID":"75485640-6686-42ef-b6fe-c9834e405540","Type":"ContainerDied","Data":"36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a"}
Oct 30 03:00:39.642744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.642410    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk" event={"ID":"75485640-6686-42ef-b6fe-c9834e405540","Type":"ContainerDied","Data":"8efde8df27a40f375331b056c235d3605691438b7bae16fe2d602cc2d351f44a"}
Oct 30 03:00:39.642744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.642458    4278 scope.go:122] "RemoveContainer" containerID="36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a"
Oct 30 03:00:39.645531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.645498    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" event={"ID":"89c464f3-8c2e-4996-b4c3-7361c529384d","Type":"ContainerStarted","Data":"70ae41e5ed328618a394c03927ee98481e3f67047177c6a1ccd9b96331228e28"}
Oct 30 03:00:39.647227 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.647201    4278 generic.go:356] "Generic (PLEG): container finished" podID="1207dcbb-33f8-4776-9ea5-01f3f45ba596" containerID="c32ed659c0ed56395cb46091f45e88b96ee200927555c74447fb16339a1bc8b4" exitCode=0
Oct 30 03:00:39.647352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.647281    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l" event={"ID":"1207dcbb-33f8-4776-9ea5-01f3f45ba596","Type":"ContainerDied","Data":"c32ed659c0ed56395cb46091f45e88b96ee200927555c74447fb16339a1bc8b4"}
Oct 30 03:00:39.648928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.648904    4278 generic.go:356] "Generic (PLEG): container finished" podID="da5c5135-1f96-4fa4-b97b-b8fcd82eb187" containerID="46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd" exitCode=0
Oct 30 03:00:39.649030 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.648944    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"
Oct 30 03:00:39.649030 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.648976    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw" event={"ID":"da5c5135-1f96-4fa4-b97b-b8fcd82eb187","Type":"ContainerDied","Data":"46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd"}
Oct 30 03:00:39.649030 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.649018    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw" event={"ID":"da5c5135-1f96-4fa4-b97b-b8fcd82eb187","Type":"ContainerDied","Data":"b7b53f84a350ae3c6d3c597f87c15649d3c9e3e8d2ed77ea790d2408a33dc16d"}
Oct 30 03:00:39.650643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.650625    4278 generic.go:356] "Generic (PLEG): container finished" podID="656f02c8-4d75-42e4-8600-0861f87e2828" containerID="ab456413fd34d7422604d5aef3a0b5e4bbadf60bd590e763729039bdef04f75e" exitCode=0
Oct 30 03:00:39.650750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.650684    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr" event={"ID":"656f02c8-4d75-42e4-8600-0861f87e2828","Type":"ContainerDied","Data":"ab456413fd34d7422604d5aef3a0b5e4bbadf60bd590e763729039bdef04f75e"}
Oct 30 03:00:39.655903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.655884    4278 scope.go:122] "RemoveContainer" containerID="36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a"
Oct 30 03:00:39.656330 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:39.656303    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a\": container with ID starting with 36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a not found: ID does not exist" containerID="36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a"
Oct 30 03:00:39.656510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.656341    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a"} err="failed to get container status \"36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a\": rpc error: code = NotFound desc = could not find container \"36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a\": container with ID starting with 36de3c9da9d7303a1101cae7d75e7c611a23cf9977ce1563f56444b72215e06a not found: ID does not exist"
Oct 30 03:00:39.656510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.656363    4278 scope.go:122] "RemoveContainer" containerID="46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd"
Oct 30 03:00:39.691471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.691443    4278 scope.go:122] "RemoveContainer" containerID="46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd"
Oct 30 03:00:39.692138 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:39.692065    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd\": container with ID starting with 46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd not found: ID does not exist" containerID="46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd"
Oct 30 03:00:39.692138 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.692105    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd"} err="failed to get container status \"46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd\": rpc error: code = NotFound desc = could not find container \"46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd\": container with ID starting with 46a3a90a4ba79f80084b75864b928ca91d21fa29799803206ab75aa01bb1e2dd not found: ID does not exist"
Oct 30 03:00:39.909811 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.909777    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"]
Oct 30 03:00:39.916489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.916464    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2b0506d-655c4bdfdb-kx6mk"]
Oct 30 03:00:39.985516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.981669    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"]
Oct 30 03:00:39.998689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:39.998651    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2b0506d-7544d9c559-n49vw"]
Oct 30 03:00:40.031780 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:40.031734    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"]
Oct 30 03:00:40.035953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:40.035922    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20dc1f04-gjjkj"]
Oct 30 03:00:40.202089 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:40.202049    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75485640-6686-42ef-b6fe-c9834e405540" path="/var/lib/kubelet/pods/75485640-6686-42ef-b6fe-c9834e405540/volumes"
Oct 30 03:00:40.202644 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:40.202621    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da5c5135-1f96-4fa4-b97b-b8fcd82eb187" path="/var/lib/kubelet/pods/da5c5135-1f96-4fa4-b97b-b8fcd82eb187/volumes"
Oct 30 03:00:40.203096 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:40.203077    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1b8fb66-3018-4809-bb20-688f79c63ef8" path="/var/lib/kubelet/pods/e1b8fb66-3018-4809-bb20-688f79c63ef8/volumes"
Oct 30 03:00:40.477679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:40.475046    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 03:00:40.485463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:40.484561    4278 patch_prober.go:29] interesting pod/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 03:00:40.485463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:40.484624    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 03:00:40.902247 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:40.900413    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:40.948172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:40.948145    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:41.003578 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.003494    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-openshift-ca\") pod \"656f02c8-4d75-42e4-8600-0861f87e2828\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") "
Oct 30 03:00:41.003578 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.003539    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/656f02c8-4d75-42e4-8600-0861f87e2828-kube-api-access-69rnl\" (UniqueName: \"kubernetes.io/projected/656f02c8-4d75-42e4-8600-0861f87e2828-kube-api-access-69rnl\") pod \"656f02c8-4d75-42e4-8600-0861f87e2828\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") "
Oct 30 03:00:41.003794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.003600    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1207dcbb-33f8-4776-9ea5-01f3f45ba596-kube-api-access-5tjm8\" (UniqueName: \"kubernetes.io/projected/1207dcbb-33f8-4776-9ea5-01f3f45ba596-kube-api-access-5tjm8\") pod \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") "
Oct 30 03:00:41.003794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.003683    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-openshift-ca\") pod \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") "
Oct 30 03:00:41.003910 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.003875    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-openshift-ca" pod "656f02c8-4d75-42e4-8600-0861f87e2828" (UID: "656f02c8-4d75-42e4-8600-0861f87e2828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:41.003968 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.003905    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-xks-ca\") pod \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\" (UID: \"1207dcbb-33f8-4776-9ea5-01f3f45ba596\") "
Oct 30 03:00:41.004024 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.004013    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-xks-ca\") pod \"656f02c8-4d75-42e4-8600-0861f87e2828\" (UID: \"656f02c8-4d75-42e4-8600-0861f87e2828\") "
Oct 30 03:00:41.004122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.004098    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-openshift-ca" pod "1207dcbb-33f8-4776-9ea5-01f3f45ba596" (UID: "1207dcbb-33f8-4776-9ea5-01f3f45ba596"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:41.004737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.004710    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:41.004865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.004743    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:41.006240 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.006215    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-xks-ca" pod "656f02c8-4d75-42e4-8600-0861f87e2828" (UID: "656f02c8-4d75-42e4-8600-0861f87e2828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:41.006697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.006652    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656f02c8-4d75-42e4-8600-0861f87e2828-kube-api-access-69rnl" pod "656f02c8-4d75-42e4-8600-0861f87e2828" (UID: "656f02c8-4d75-42e4-8600-0861f87e2828"). InnerVolumeSpecName "kube-api-access-69rnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:41.006792 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.006733    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1207dcbb-33f8-4776-9ea5-01f3f45ba596-kube-api-access-5tjm8" pod "1207dcbb-33f8-4776-9ea5-01f3f45ba596" (UID: "1207dcbb-33f8-4776-9ea5-01f3f45ba596"). InnerVolumeSpecName "kube-api-access-5tjm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:41.006868 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.006808    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-xks-ca" pod "1207dcbb-33f8-4776-9ea5-01f3f45ba596" (UID: "1207dcbb-33f8-4776-9ea5-01f3f45ba596"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:41.106647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.106611    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1207dcbb-33f8-4776-9ea5-01f3f45ba596-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:41.106647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.106639    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656f02c8-4d75-42e4-8600-0861f87e2828-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:41.106647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.106648    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69rnl\" (UniqueName: \"kubernetes.io/projected/656f02c8-4d75-42e4-8600-0861f87e2828-kube-api-access-69rnl\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:41.106647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.106658    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tjm8\" (UniqueName: \"kubernetes.io/projected/1207dcbb-33f8-4776-9ea5-01f3f45ba596-kube-api-access-5tjm8\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:41.483451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.481824    4278 patch_prober.go:29] interesting pod/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 03:00:41.483451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.481887    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 03:00:41.694287 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.694261    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"
Oct 30 03:00:41.694486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.694257    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l" event={"ID":"1207dcbb-33f8-4776-9ea5-01f3f45ba596","Type":"ContainerDied","Data":"32ba6e2262ee270d3a33f013e1f6dac1bf6f06f9a48da26f18b651c22197b1a7"}
Oct 30 03:00:41.694486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.694412    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32ba6e2262ee270d3a33f013e1f6dac1bf6f06f9a48da26f18b651c22197b1a7"
Oct 30 03:00:41.695923 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.695906    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"
Oct 30 03:00:41.696035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.695947    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr" event={"ID":"656f02c8-4d75-42e4-8600-0861f87e2828","Type":"ContainerDied","Data":"48857db74af0d3b6b13e831a22483d4f153c1886af7bd4829792144ad192c809"}
Oct 30 03:00:41.696035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:41.695992    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48857db74af0d3b6b13e831a22483d4f153c1886af7bd4829792144ad192c809"
Oct 30 03:00:42.480448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:42.479839    4278 patch_prober.go:29] interesting pod/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 03:00:42.480448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:42.479901    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 03:00:43.483552 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:43.483513    4278 patch_prober.go:29] interesting pod/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 03:00:43.484052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:43.483591    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 03:00:43.692898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:43.692838    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4" podUID="f4dee307-b518-4950-93b3-c99f2551ddce" containerName="payload-processing" containerID="cri-o://fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba" gracePeriod=25
Oct 30 03:00:44.032496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.031069    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"]
Oct 30 03:00:44.053843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.053802    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e78d35-thq59"]
Oct 30 03:00:44.166500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.163647    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 03:00:44.264447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.259777    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dee307-b518-4950-93b3-c99f2551ddce-kube-api-access-hk7b8\" (UniqueName: \"kubernetes.io/projected/f4dee307-b518-4950-93b3-c99f2551ddce-kube-api-access-hk7b8\") pod \"f4dee307-b518-4950-93b3-c99f2551ddce\" (UID: \"f4dee307-b518-4950-93b3-c99f2551ddce\") "
Oct 30 03:00:44.264447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.259870    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dee307-b518-4950-93b3-c99f2551ddce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dee307-b518-4950-93b3-c99f2551ddce-plugins-config-volume\") pod \"f4dee307-b518-4950-93b3-c99f2551ddce\" (UID: \"f4dee307-b518-4950-93b3-c99f2551ddce\") "
Oct 30 03:00:44.274440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.273824    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dee307-b518-4950-93b3-c99f2551ddce-plugins-config-volume" pod "f4dee307-b518-4950-93b3-c99f2551ddce" (UID: "f4dee307-b518-4950-93b3-c99f2551ddce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:44.320776 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.320738    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8417cff-2dcc-44c2-bd63-edaa5a4b6679" path="/var/lib/kubelet/pods/c8417cff-2dcc-44c2-bd63-edaa5a4b6679/volumes"
Oct 30 03:00:44.326857 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.326806    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dee307-b518-4950-93b3-c99f2551ddce-kube-api-access-hk7b8" pod "f4dee307-b518-4950-93b3-c99f2551ddce" (UID: "f4dee307-b518-4950-93b3-c99f2551ddce"). InnerVolumeSpecName "kube-api-access-hk7b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:44.369799 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.369766    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk7b8\" (UniqueName: \"kubernetes.io/projected/f4dee307-b518-4950-93b3-c99f2551ddce-kube-api-access-hk7b8\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:44.370029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.370008    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dee307-b518-4950-93b3-c99f2551ddce-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:44.485510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.480053    4278 patch_prober.go:29] interesting pod/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 03:00:44.485510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.480114    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 03:00:44.741227 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.741135    4278 generic.go:356] "Generic (PLEG): container finished" podID="f4dee307-b518-4950-93b3-c99f2551ddce" containerID="fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba" exitCode=0
Oct 30 03:00:44.741671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.741468    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4" event={"ID":"f4dee307-b518-4950-93b3-c99f2551ddce","Type":"ContainerDied","Data":"fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba"}
Oct 30 03:00:44.742314 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.742295    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4" event={"ID":"f4dee307-b518-4950-93b3-c99f2551ddce","Type":"ContainerDied","Data":"8a84d2d03a9631530cf0e168eefd33d827c6c151adfcfd6aaff08d2731aac409"}
Oct 30 03:00:44.742497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.742482    4278 scope.go:122] "RemoveContainer" containerID="fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba"
Oct 30 03:00:44.742742 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.741539    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"
Oct 30 03:00:44.792759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.792729    4278 scope.go:122] "RemoveContainer" containerID="fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba"
Oct 30 03:00:44.793288 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:44.793265    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba\": container with ID starting with fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba not found: ID does not exist" containerID="fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba"
Oct 30 03:00:44.793413 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.793300    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba"} err="failed to get container status \"fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba\": rpc error: code = NotFound desc = could not find container \"fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba\": container with ID starting with fff2929331fe8c4409f73ef0331549599903770cf3c898511aa331b38513cfba not found: ID does not exist"
Oct 30 03:00:44.808680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.808640    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"]
Oct 30 03:00:44.821472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.821450    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93fe9de7-55ffd7cd84-ms4k4"]
Oct 30 03:00:44.984648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.984598    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"]
Oct 30 03:00:44.989824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:44.989792    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2b0506d-99d9d"]
Oct 30 03:00:45.475224 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:45.475193    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 03:00:45.475665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:45.475644    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 03:00:45.476685 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:45.476661    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 03:00:46.205154 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:46.205118    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79530fd5-67b0-45b1-ba8a-8c271cd44fd6" path="/var/lib/kubelet/pods/79530fd5-67b0-45b1-ba8a-8c271cd44fd6/volumes"
Oct 30 03:00:46.205671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:46.205654    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dee307-b518-4950-93b3-c99f2551ddce" path="/var/lib/kubelet/pods/f4dee307-b518-4950-93b3-c99f2551ddce/volumes"
Oct 30 03:00:46.682878 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:46.682837    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:00:48.325758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:48.323535    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"]
Oct 30 03:00:49.538450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:49.514523    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"]
Oct 30 03:00:49.691744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:49.691696    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93fe9de7-r5rmb"]
Oct 30 03:00:50.223211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.223169    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253b29df-ffcc-4d70-9c91-4954b5c28be7" path="/var/lib/kubelet/pods/253b29df-ffcc-4d70-9c91-4954b5c28be7/volumes"
Oct 30 03:00:50.498796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.498706    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"]
Oct 30 03:00:50.499468 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499434    4278 cpu_manager.go:414] "containerMap: removing container" podUID="1207dcbb-33f8-4776-9ea5-01f3f45ba596" containerName="revoke-keys"
Oct 30 03:00:50.499468 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499457    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1207dcbb-33f8-4776-9ea5-01f3f45ba596" containerName="revoke-keys"
Oct 30 03:00:50.499682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499510    4278 cpu_manager.go:414] "containerMap: removing container" podUID="da5c5135-1f96-4fa4-b97b-b8fcd82eb187" containerName="payload-pre-processing"
Oct 30 03:00:50.499682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499520    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5c5135-1f96-4fa4-b97b-b8fcd82eb187" containerName="payload-pre-processing"
Oct 30 03:00:50.499682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499554    4278 cpu_manager.go:414] "containerMap: removing container" podUID="656f02c8-4d75-42e4-8600-0861f87e2828" containerName="revoke-keys"
Oct 30 03:00:50.499682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499563    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656f02c8-4d75-42e4-8600-0861f87e2828" containerName="revoke-keys"
Oct 30 03:00:50.499682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499578    4278 cpu_manager.go:414] "containerMap: removing container" podUID="75485640-6686-42ef-b6fe-c9834e405540" containerName="payload-processing"
Oct 30 03:00:50.499682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499585    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75485640-6686-42ef-b6fe-c9834e405540" containerName="payload-processing"
Oct 30 03:00:50.499682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499597    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f4dee307-b518-4950-93b3-c99f2551ddce" containerName="payload-processing"
Oct 30 03:00:50.499682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499605    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dee307-b518-4950-93b3-c99f2551ddce" containerName="payload-processing"
Oct 30 03:00:50.500062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499744    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="656f02c8-4d75-42e4-8600-0861f87e2828" containerName="revoke-keys"
Oct 30 03:00:50.500062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499759    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="75485640-6686-42ef-b6fe-c9834e405540" containerName="payload-processing"
Oct 30 03:00:50.500062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499769    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dee307-b518-4950-93b3-c99f2551ddce" containerName="payload-processing"
Oct 30 03:00:50.500062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499779    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="1207dcbb-33f8-4776-9ea5-01f3f45ba596" containerName="revoke-keys"
Oct 30 03:00:50.500062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.499794    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5c5135-1f96-4fa4-b97b-b8fcd82eb187" containerName="payload-pre-processing"
Oct 30 03:00:50.503636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.503614    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.530560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.529233    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"]
Oct 30 03:00:50.536680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.536649    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"distinct-w1-e11f03-kserve-self-signed-certs\""
Oct 30 03:00:50.560970 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.560932    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-model-cache\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.566131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.566098    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnprh\" (UniqueName: \"kubernetes.io/projected/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-kube-api-access-xnprh\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.566514 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.566489    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tmp-dir\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.566675 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.566658    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-dshm\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.566812 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.566785    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-home\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.567009 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.566989    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tls-certs\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.668161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.668125    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-model-cache\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.668373 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.668198    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnprh\" (UniqueName: \"kubernetes.io/projected/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-kube-api-access-xnprh\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.668373 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.668359    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tmp-dir\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.668513 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.668397    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-dshm\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.668513 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.668453    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-home\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.668619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.668541    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tls-certs\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.669337 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.669041    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tmp-dir\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.669337 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.669281    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-home\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.669337 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.669301    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-model-cache\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.673793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.673735    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-dshm\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.674052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.673989    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tls-certs\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.695670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.695639    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnprh\" (UniqueName: \"kubernetes.io/projected/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-kube-api-access-xnprh\") pod \"distinct-w1-e11f03-kserve-5fbbd4f78d-skr54\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:50.819779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:50.819601    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:51.240154 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:51.240120    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0a55e03_c9d8_4ef8_bdf3_3df6495cb327.slice/crio-795c51676f5e1f97d7e339b77e6ae871406f3aeb11c4cb2804d303f477dca167 WatchSource:0}: Error finding container 795c51676f5e1f97d7e339b77e6ae871406f3aeb11c4cb2804d303f477dca167: Status 404 returned error can't find the container with id 795c51676f5e1f97d7e339b77e6ae871406f3aeb11c4cb2804d303f477dca167
Oct 30 03:00:51.248519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:51.246054    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"]
Oct 30 03:00:51.916669 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:51.916625    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" event={"ID":"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327","Type":"ContainerStarted","Data":"820becf7904a6452ec59e8a953ad37706befcfda59b68a68c771deb8440ba193"}
Oct 30 03:00:51.917108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:51.916692    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" event={"ID":"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327","Type":"ContainerStarted","Data":"795c51676f5e1f97d7e339b77e6ae871406f3aeb11c4cb2804d303f477dca167"}
Oct 30 03:00:52.820998 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:52.820966    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:00:53.324100 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.324059    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42" podUID="74ad723e-7a76-4fe1-985e-8cd0149b7f60" containerName="payload-processing" containerID="cri-o://b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2" gracePeriod=25
Oct 30 03:00:53.603692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.603642    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 03:00:53.657991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.657886    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" podStartSLOduration=3.657861958 podStartE2EDuration="3.657861958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:51.961561207 +0000 UTC m=+1189.885769406" watchObservedRunningTime="2026-10-30 03:00:53.657861958 +0000 UTC m=+1191.582070147"
Oct 30 03:00:53.740023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.739987    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ad723e-7a76-4fe1-985e-8cd0149b7f60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ad723e-7a76-4fe1-985e-8cd0149b7f60-plugins-config-volume\") pod \"74ad723e-7a76-4fe1-985e-8cd0149b7f60\" (UID: \"74ad723e-7a76-4fe1-985e-8cd0149b7f60\") "
Oct 30 03:00:53.740195 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.740154    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ad723e-7a76-4fe1-985e-8cd0149b7f60-kube-api-access-v56ws\" (UniqueName: \"kubernetes.io/projected/74ad723e-7a76-4fe1-985e-8cd0149b7f60-kube-api-access-v56ws\") pod \"74ad723e-7a76-4fe1-985e-8cd0149b7f60\" (UID: \"74ad723e-7a76-4fe1-985e-8cd0149b7f60\") "
Oct 30 03:00:53.740519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.740485    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ad723e-7a76-4fe1-985e-8cd0149b7f60-plugins-config-volume" pod "74ad723e-7a76-4fe1-985e-8cd0149b7f60" (UID: "74ad723e-7a76-4fe1-985e-8cd0149b7f60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:53.740744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.740724    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ad723e-7a76-4fe1-985e-8cd0149b7f60-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:53.743169 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.743136    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ad723e-7a76-4fe1-985e-8cd0149b7f60-kube-api-access-v56ws" pod "74ad723e-7a76-4fe1-985e-8cd0149b7f60" (UID: "74ad723e-7a76-4fe1-985e-8cd0149b7f60"). InnerVolumeSpecName "kube-api-access-v56ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:53.842793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.842710    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v56ws\" (UniqueName: \"kubernetes.io/projected/74ad723e-7a76-4fe1-985e-8cd0149b7f60-kube-api-access-v56ws\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:00:53.909656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.903318    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"]
Oct 30 03:00:53.909656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.909010    4278 cpu_manager.go:414] "containerMap: removing container" podUID="74ad723e-7a76-4fe1-985e-8cd0149b7f60" containerName="payload-processing"
Oct 30 03:00:53.909656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.909046    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ad723e-7a76-4fe1-985e-8cd0149b7f60" containerName="payload-processing"
Oct 30 03:00:53.909656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.909456    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ad723e-7a76-4fe1-985e-8cd0149b7f60" containerName="payload-processing"
Oct 30 03:00:53.913892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.913865    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:00:53.927336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.927133    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"]
Oct 30 03:00:53.935257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.930932    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9ca6fbc2\""
Oct 30 03:00:53.935257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.931006    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9ca6fbc2-dockercfg-wvg9l\""
Oct 30 03:00:53.939788 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.939757    4278 generic.go:356] "Generic (PLEG): container finished" podID="74ad723e-7a76-4fe1-985e-8cd0149b7f60" containerID="b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2" exitCode=0
Oct 30 03:00:53.939922 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.939846    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"
Oct 30 03:00:53.940219 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.940180    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42" event={"ID":"74ad723e-7a76-4fe1-985e-8cd0149b7f60","Type":"ContainerDied","Data":"b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2"}
Oct 30 03:00:53.940376 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.940359    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42" event={"ID":"74ad723e-7a76-4fe1-985e-8cd0149b7f60","Type":"ContainerDied","Data":"b627f5b52c155fd6940b104e90e190214136a0c8a6f98d83049fa5bdbac334b2"}
Oct 30 03:00:53.940549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.940519    4278 scope.go:122] "RemoveContainer" containerID="b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2"
Oct 30 03:00:53.955648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.955258    4278 scope.go:122] "RemoveContainer" containerID="b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2"
Oct 30 03:00:53.955728 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:53.955701    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2\": container with ID starting with b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2 not found: ID does not exist" containerID="b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2"
Oct 30 03:00:53.955786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:53.955734    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2"} err="failed to get container status \"b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2\": rpc error: code = NotFound desc = could not find container \"b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2\": container with ID starting with b8bd8a7ec937ceb281968b334fc560b81e5120f1b2982bf77f547bd62e77c2e2 not found: ID does not exist"
Oct 30 03:00:54.052549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.050558    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de19764-c41a-4256-a0c4-34d729e69b3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj\" (UID: \"2de19764-c41a-4256-a0c4-34d729e69b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:00:54.052549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.050631    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4cs\" (UniqueName: \"kubernetes.io/projected/2de19764-c41a-4256-a0c4-34d729e69b3c-kube-api-access-dh4cs\") pod \"payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj\" (UID: \"2de19764-c41a-4256-a0c4-34d729e69b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:00:54.074494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.071516    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"]
Oct 30 03:00:54.080743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.079583    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-adca906b-565c86788b-klv42"]
Oct 30 03:00:54.085516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.084969    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"]
Oct 30 03:00:54.089877 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.089845    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc52d0f2-9ltgb"]
Oct 30 03:00:54.151879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.151785    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de19764-c41a-4256-a0c4-34d729e69b3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj\" (UID: \"2de19764-c41a-4256-a0c4-34d729e69b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:00:54.151879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.151832    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4cs\" (UniqueName: \"kubernetes.io/projected/2de19764-c41a-4256-a0c4-34d729e69b3c-kube-api-access-dh4cs\") pod \"payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj\" (UID: \"2de19764-c41a-4256-a0c4-34d729e69b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:00:54.152551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.152529    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de19764-c41a-4256-a0c4-34d729e69b3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj\" (UID: \"2de19764-c41a-4256-a0c4-34d729e69b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:00:54.189054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.189007    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4cs\" (UniqueName: \"kubernetes.io/projected/2de19764-c41a-4256-a0c4-34d729e69b3c-kube-api-access-dh4cs\") pod \"payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj\" (UID: \"2de19764-c41a-4256-a0c4-34d729e69b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:00:54.207553 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.207001    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6fbe18-5093-4e5a-8545-50f22759f44f" path="/var/lib/kubelet/pods/5b6fbe18-5093-4e5a-8545-50f22759f44f/volumes"
Oct 30 03:00:54.208552 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.208530    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ad723e-7a76-4fe1-985e-8cd0149b7f60" path="/var/lib/kubelet/pods/74ad723e-7a76-4fe1-985e-8cd0149b7f60/volumes"
Oct 30 03:00:54.238357 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.238066    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:00:54.294218 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:00:54.294180    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:00:54.662838 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:00:54.662805    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de19764_c41a_4256_a0c4_34d729e69b3c.slice/crio-8181d28d8a754a9efb4a6f8da1dd2f87a67ab4139da2ec82335cc8683e5cb2a2 WatchSource:0}: Error finding container 8181d28d8a754a9efb4a6f8da1dd2f87a67ab4139da2ec82335cc8683e5cb2a2: Status 404 returned error can't find the container with id 8181d28d8a754a9efb4a6f8da1dd2f87a67ab4139da2ec82335cc8683e5cb2a2
Oct 30 03:00:54.664485 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.664453    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"]
Oct 30 03:00:54.951768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.951644    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj" event={"ID":"2de19764-c41a-4256-a0c4-34d729e69b3c","Type":"ContainerStarted","Data":"1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4"}
Oct 30 03:00:54.951768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.951724    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj" event={"ID":"2de19764-c41a-4256-a0c4-34d729e69b3c","Type":"ContainerStarted","Data":"8181d28d8a754a9efb4a6f8da1dd2f87a67ab4139da2ec82335cc8683e5cb2a2"}
Oct 30 03:00:54.951768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.951757    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:00:54.996211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:54.996048    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj" podStartSLOduration=1.99602815 podStartE2EDuration="1.99602815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:54.993500359 +0000 UTC m=+1192.917708572" watchObservedRunningTime="2026-10-30 03:00:54.99602815 +0000 UTC m=+1192.920236339"
Oct 30 03:00:55.320033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:55.319992    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"]
Oct 30 03:00:55.325661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:55.325623    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-adca906b-5fx7l"]
Oct 30 03:00:56.201849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:56.201815    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1207dcbb-33f8-4776-9ea5-01f3f45ba596" path="/var/lib/kubelet/pods/1207dcbb-33f8-4776-9ea5-01f3f45ba596/volumes"
Oct 30 03:00:56.754208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:56.754175    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"]
Oct 30 03:00:56.754491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:00:56.754462    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy" containerID="cri-o://70ae41e5ed328618a394c03927ee98481e3f67047177c6a1ccd9b96331228e28" gracePeriod=30
Oct 30 03:01:00.477805 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:00.477768    4278 patch_prober.go:29] interesting pod/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:01:00.478162 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:00.477849    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:01:00.820915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:00.820883    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:01:00.833965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:00.833922    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:01:01.028997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.028965    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:01:01.280228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.280183    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"]
Oct 30 03:01:01.315446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.311731    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:01:01.341503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.335368    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-9245d1\""
Oct 30 03:01:01.341503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.335749    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-9245d1-dockercfg-w7xvw\""
Oct 30 03:01:01.403993 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.403947    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82t7\" (UniqueName: \"kubernetes.io/projected/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-kube-api-access-b82t7\") pod \"payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7\" (UID: \"9f865b44-0a96-4d11-a201-ed6dd3a6ade0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:01:01.404379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.404354    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7\" (UID: \"9f865b44-0a96-4d11-a201-ed6dd3a6ade0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:01:01.445866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.445820    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"]
Oct 30 03:01:01.515945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.515891    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82t7\" (UniqueName: \"kubernetes.io/projected/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-kube-api-access-b82t7\") pod \"payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7\" (UID: \"9f865b44-0a96-4d11-a201-ed6dd3a6ade0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:01:01.516410 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.516003    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7\" (UID: \"9f865b44-0a96-4d11-a201-ed6dd3a6ade0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:01:01.516890 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.516862    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7\" (UID: \"9f865b44-0a96-4d11-a201-ed6dd3a6ade0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:01:01.536817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.536732    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82t7\" (UniqueName: \"kubernetes.io/projected/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-kube-api-access-b82t7\") pod \"payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7\" (UID: \"9f865b44-0a96-4d11-a201-ed6dd3a6ade0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:01:01.734730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:01.734695    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:01:02.025741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.025708    4278 generic.go:356] "Generic (PLEG): container finished" podID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerID="70ae41e5ed328618a394c03927ee98481e3f67047177c6a1ccd9b96331228e28" exitCode=0
Oct 30 03:01:02.025925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.025786    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" event={"ID":"89c464f3-8c2e-4996-b4c3-7361c529384d","Type":"ContainerDied","Data":"70ae41e5ed328618a394c03927ee98481e3f67047177c6a1ccd9b96331228e28"}
Oct 30 03:01:02.025925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.025866    4278 scope.go:122] "RemoveContainer" containerID="f9a4436cc155887b6d8b4dd94768cb3c0e9dfc121e7247e9dd37a4c03c7e2f03"
Oct 30 03:01:02.041805 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.041780    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 03:01:02.058069 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:01:02.058037    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f865b44_0a96_4d11_a201_ed6dd3a6ade0.slice/crio-25e1555f522be57d096fc0416201dae3d71766dd1789922c1428361b02aac90c WatchSource:0}: Error finding container 25e1555f522be57d096fc0416201dae3d71766dd1789922c1428361b02aac90c: Status 404 returned error can't find the container with id 25e1555f522be57d096fc0416201dae3d71766dd1789922c1428361b02aac90c
Oct 30 03:01:02.058761 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.058734    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"]
Oct 30 03:01:02.126869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.126841    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c464f3-8c2e-4996-b4c3-7361c529384d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c464f3-8c2e-4996-b4c3-7361c529384d-istiod-ca-cert\") pod \"89c464f3-8c2e-4996-b4c3-7361c529384d\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") "
Oct 30 03:01:02.126949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.126896    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-data\") pod \"89c464f3-8c2e-4996-b4c3-7361c529384d\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") "
Oct 30 03:01:02.127014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.126971    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-certs\") pod \"89c464f3-8c2e-4996-b4c3-7361c529384d\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") "
Oct 30 03:01:02.127014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.127004    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-podinfo\") pod \"89c464f3-8c2e-4996-b4c3-7361c529384d\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") "
Oct 30 03:01:02.127118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.127066    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-token\" (UniqueName: \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-token\") pod \"89c464f3-8c2e-4996-b4c3-7361c529384d\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") "
Oct 30 03:01:02.127175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.127165    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-credential-socket\") pod \"89c464f3-8c2e-4996-b4c3-7361c529384d\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") "
Oct 30 03:01:02.127235 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.127171    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-data" pod "89c464f3-8c2e-4996-b4c3-7361c529384d" (UID: "89c464f3-8c2e-4996-b4c3-7361c529384d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:01:02.127235 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.127200    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-kube-api-access-7ptcd\" (UniqueName: \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-kube-api-access-7ptcd\") pod \"89c464f3-8c2e-4996-b4c3-7361c529384d\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") "
Oct 30 03:01:02.127335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.127290    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-envoy\") pod \"89c464f3-8c2e-4996-b4c3-7361c529384d\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") "
Oct 30 03:01:02.127335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.127321    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-socket\") pod \"89c464f3-8c2e-4996-b4c3-7361c529384d\" (UID: \"89c464f3-8c2e-4996-b4c3-7361c529384d\") "
Oct 30 03:01:02.127335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.127324    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c464f3-8c2e-4996-b4c3-7361c529384d-istiod-ca-cert" pod "89c464f3-8c2e-4996-b4c3-7361c529384d" (UID: "89c464f3-8c2e-4996-b4c3-7361c529384d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:01:02.127578 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.127560    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-certs" pod "89c464f3-8c2e-4996-b4c3-7361c529384d" (UID: "89c464f3-8c2e-4996-b4c3-7361c529384d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:01:02.127682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.127651    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-credential-socket" pod "89c464f3-8c2e-4996-b4c3-7361c529384d" (UID: "89c464f3-8c2e-4996-b4c3-7361c529384d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:01:02.128101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.128072    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-socket" pod "89c464f3-8c2e-4996-b4c3-7361c529384d" (UID: "89c464f3-8c2e-4996-b4c3-7361c529384d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:01:02.128198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.128086    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c464f3-8c2e-4996-b4c3-7361c529384d-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:02.128198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.128127    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:02.128198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.128143    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:02.128198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.128158    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:02.129919 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.129884    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-podinfo" pod "89c464f3-8c2e-4996-b4c3-7361c529384d" (UID: "89c464f3-8c2e-4996-b4c3-7361c529384d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:01:02.130237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.130211    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-envoy" pod "89c464f3-8c2e-4996-b4c3-7361c529384d" (UID: "89c464f3-8c2e-4996-b4c3-7361c529384d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:01:02.130667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.130641    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-kube-api-access-7ptcd" pod "89c464f3-8c2e-4996-b4c3-7361c529384d" (UID: "89c464f3-8c2e-4996-b4c3-7361c529384d"). InnerVolumeSpecName "kube-api-access-7ptcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:01:02.130930 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.130908    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-token" pod "89c464f3-8c2e-4996-b4c3-7361c529384d" (UID: "89c464f3-8c2e-4996-b4c3-7361c529384d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:01:02.228970 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.228940    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:02.228970 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.228967    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:02.228970 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.228978    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ptcd\" (UniqueName: \"kubernetes.io/projected/89c464f3-8c2e-4996-b4c3-7361c529384d-kube-api-access-7ptcd\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:02.229180 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.228987    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:02.229180 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.228996    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c464f3-8c2e-4996-b4c3-7361c529384d-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:02.378363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:02.378265    4278 scope.go:122] "RemoveContainer" containerID="e2d46f7a93f95236073f739060ef6cb68113e55ed369691d858ede1355b8d66d"
Oct 30 03:01:03.034471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:03.034412    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7" event={"ID":"9f865b44-0a96-4d11-a201-ed6dd3a6ade0","Type":"ContainerStarted","Data":"10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960"}
Oct 30 03:01:03.034916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:03.034541    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:01:03.034916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:03.034569    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7" event={"ID":"9f865b44-0a96-4d11-a201-ed6dd3a6ade0","Type":"ContainerStarted","Data":"25e1555f522be57d096fc0416201dae3d71766dd1789922c1428361b02aac90c"}
Oct 30 03:01:03.036083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:03.036061    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"
Oct 30 03:01:03.036178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:03.036090    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt" event={"ID":"89c464f3-8c2e-4996-b4c3-7361c529384d","Type":"ContainerDied","Data":"bd0ad495aebb5af0b2cccfb4d998634ce92324cfb7d7631d00111b48a7e7b392"}
Oct 30 03:01:03.036178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:03.036152    4278 scope.go:122] "RemoveContainer" containerID="70ae41e5ed328618a394c03927ee98481e3f67047177c6a1ccd9b96331228e28"
Oct 30 03:01:03.080625 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:03.080570    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7" podStartSLOduration=2.080554045 podStartE2EDuration="2.080554045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:03.078453943 +0000 UTC m=+1201.002662122" watchObservedRunningTime="2026-10-30 03:01:03.080554045 +0000 UTC m=+1201.004762233"
Oct 30 03:01:03.147795 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:03.147761    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"]
Oct 30 03:01:03.157750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:03.157718    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-adca906b-openshift-default-6dc6cc78d5-jzrjt"]
Oct 30 03:01:04.202421 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:04.202383    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" path="/var/lib/kubelet/pods/89c464f3-8c2e-4996-b4c3-7361c529384d/volumes"
Oct 30 03:01:05.989483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:05.987761    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:01:14.048088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:14.048059    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:01:15.235877 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.235796    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"]
Oct 30 03:01:15.236712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.236681    4278 cpu_manager.go:414] "containerMap: removing container" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy"
Oct 30 03:01:15.236712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.236710    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy"
Oct 30 03:01:15.236836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.236760    4278 cpu_manager.go:414] "containerMap: removing container" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy"
Oct 30 03:01:15.236836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.236770    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy"
Oct 30 03:01:15.236949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.236938    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy"
Oct 30 03:01:15.236983 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.236959    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c464f3-8c2e-4996-b4c3-7361c529384d" containerName="istio-proxy"
Oct 30 03:01:15.242022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.241999    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.257346 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.257324    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"]
Oct 30 03:01:15.280443 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.273811    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-9245d1-serving-cert\""
Oct 30 03:01:15.343509 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.343470    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a3fdf8-344b-4be7-a56b-6edd1489b992-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.343697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.343537    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.343760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.343688    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.343854 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.343830    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4t2r\" (UniqueName: \"kubernetes.io/projected/a0a3fdf8-344b-4be7-a56b-6edd1489b992-kube-api-access-c4t2r\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.444975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.444940    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4t2r\" (UniqueName: \"kubernetes.io/projected/a0a3fdf8-344b-4be7-a56b-6edd1489b992-kube-api-access-c4t2r\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.445202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.445129    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a3fdf8-344b-4be7-a56b-6edd1489b992-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.445202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.445172    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.445324 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.445274    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.445916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.445892    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a3fdf8-344b-4be7-a56b-6edd1489b992-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.447806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.447783    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.448004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.447987    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.466989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.466957    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4t2r\" (UniqueName: \"kubernetes.io/projected/a0a3fdf8-344b-4be7-a56b-6edd1489b992-kube-api-access-c4t2r\") pod \"maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.557970 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.557939    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:15.874867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.874836    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"]
Oct 30 03:01:15.877333 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:01:15.877303    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a3fdf8_344b_4be7_a56b_6edd1489b992.slice/crio-b877b8dfe6aeb6f0a4e28e9f03745e79432266ebefef72963ec5f2a67c565c07 WatchSource:0}: Error finding container b877b8dfe6aeb6f0a4e28e9f03745e79432266ebefef72963ec5f2a67c565c07: Status 404 returned error can't find the container with id b877b8dfe6aeb6f0a4e28e9f03745e79432266ebefef72963ec5f2a67c565c07
Oct 30 03:01:15.879108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:15.879082    4278 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:01:16.170347 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.170237    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h" event={"ID":"a0a3fdf8-344b-4be7-a56b-6edd1489b992","Type":"ContainerStarted","Data":"b877b8dfe6aeb6f0a4e28e9f03745e79432266ebefef72963ec5f2a67c565c07"}
Oct 30 03:01:16.188823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.188787    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"]
Oct 30 03:01:16.194999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.194968    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.225913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.225882    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"]
Oct 30 03:01:16.227075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.227049    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 03:01:16.227245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.227105    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 03:01:16.227245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.227208    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ctkqx\""
Oct 30 03:01:16.227245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.227246    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 03:01:16.266217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.266184    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.266662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.266319    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlhf\" (UniqueName: \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-kube-api-access-shlhf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.266662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.266353    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.266662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.266464    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.266662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.266495    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.266662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.266584    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.266662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.266638    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.368389 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.368355    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.368594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.368480    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlhf\" (UniqueName: \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-kube-api-access-shlhf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.368594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.368515    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.368594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.368543    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.368594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.368570    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.368821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.368630    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.368821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.368706    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.370012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.369978    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.371135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.371103    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.371236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.371183    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.371636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.371608    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.371889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.371866    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.387006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.386979    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlhf\" (UniqueName: \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-kube-api-access-shlhf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.387158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.387136    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mhnwk\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.525117 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.525091    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:16.858142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:16.858109    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"]
Oct 30 03:01:16.859501 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:01:16.859466    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f6743d_b3e3_42a2_b1ff_86ae3fd54985.slice/crio-ac2ca55ba37d22e1c4cd67cda6827cc1d8413c8029ad329def2d10bea240c830 WatchSource:0}: Error finding container ac2ca55ba37d22e1c4cd67cda6827cc1d8413c8029ad329def2d10bea240c830: Status 404 returned error can't find the container with id ac2ca55ba37d22e1c4cd67cda6827cc1d8413c8029ad329def2d10bea240c830
Oct 30 03:01:17.179182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:17.179099    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk" event={"ID":"88f6743d-b3e3-42a2-b1ff-86ae3fd54985","Type":"ContainerStarted","Data":"ac2ca55ba37d22e1c4cd67cda6827cc1d8413c8029ad329def2d10bea240c830"}
Oct 30 03:01:17.181105 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:17.181064    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h" event={"ID":"a0a3fdf8-344b-4be7-a56b-6edd1489b992","Type":"ContainerStarted","Data":"30f49a31badc8d413c0e1086bf17035a0af5fc4835c87572d667606cbc609d60"}
Oct 30 03:01:17.181297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:17.181276    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:17.225969 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:17.225920    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h" podStartSLOduration=1.872333791 podStartE2EDuration="2.225908076s" totalImagesPullingTime="353.574285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:17.220576097 +0000 UTC m=+1215.144784288" watchObservedRunningTime="2026-10-30 03:01:17.225908076 +0000 UTC m=+1215.150116263"
Oct 30 03:01:19.657680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:19.657643    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:19.657979 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:19.657712    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:20.208607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:20.208566    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk" event={"ID":"88f6743d-b3e3-42a2-b1ff-86ae3fd54985","Type":"ContainerStarted","Data":"7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf"}
Oct 30 03:01:20.208843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:20.208823    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:20.210707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:20.210677    4278 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mhnwk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:01:20.210837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:20.210733    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk" podUID="88f6743d-b3e3-42a2-b1ff-86ae3fd54985" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:01:20.265442 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:20.265367    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk" podStartSLOduration=1.470553272 podStartE2EDuration="4.265349258s" totalImagesPullingTime="2.794795986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:20.260995758 +0000 UTC m=+1218.185203946" watchObservedRunningTime="2026-10-30 03:01:20.265349258 +0000 UTC m=+1218.189557446"
Oct 30 03:01:21.217329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:21.217303    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:01:23.195235 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:23.195204    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:01:24.191364 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:01:24.191325    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:01:25.967486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:25.966191    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:01:28.816467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.816404    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"]
Oct 30 03:01:28.821856 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.821831    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:28.890114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.890080    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"]
Oct 30 03:01:28.984943 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.984901    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:28.985114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.984972    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:28.985114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.985066    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:28.985340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.985314    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:28.985377 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.985353    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:28.985413 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.985401    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:28.985531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.985514    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:28.985589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.985576    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:28.985624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:28.985597    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhr55\" (UniqueName: \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-kube-api-access-lhr55\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087008    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087049    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087144    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087179    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087211    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhr55\" (UniqueName: \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-kube-api-access-lhr55\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087283    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087480 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087346    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087480 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087376    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087873 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087582    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087873 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087637    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.087873 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087856    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.088071 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087892    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.088071 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.087984    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.088146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.088065    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.089564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.089544    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.089786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.089768    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.106051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.106018    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.106237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.106218    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhr55\" (UniqueName: \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-kube-api-access-lhr55\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mkrp4\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.138730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.138700    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:29.460344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.460311    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"]
Oct 30 03:01:29.462649 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:01:29.462619    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1061dc_fc7f_48e3_89ce_e1a691ab0b7f.slice/crio-d8a813e2443b10f77ddf4e5d12a888c3a3fd89d0da66e21dac37fae1d1bf9d4e WatchSource:0}: Error finding container d8a813e2443b10f77ddf4e5d12a888c3a3fd89d0da66e21dac37fae1d1bf9d4e: Status 404 returned error can't find the container with id d8a813e2443b10f77ddf4e5d12a888c3a3fd89d0da66e21dac37fae1d1bf9d4e
Oct 30 03:01:29.465463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.465412    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:29.465534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.465504    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:29.465568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:29.465538    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:30.301626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.301586    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" event={"ID":"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f","Type":"ContainerStarted","Data":"b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce"}
Oct 30 03:01:30.302072 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.301646    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" event={"ID":"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f","Type":"ContainerStarted","Data":"d8a813e2443b10f77ddf4e5d12a888c3a3fd89d0da66e21dac37fae1d1bf9d4e"}
Oct 30 03:01:30.389202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.389137    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" podStartSLOduration=2.389113098 podStartE2EDuration="2.389113098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:30.355319488 +0000 UTC m=+1228.279527688" watchObservedRunningTime="2026-10-30 03:01:30.389113098 +0000 UTC m=+1228.313321288"
Oct 30 03:01:30.516718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.510257    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"]
Oct 30 03:01:30.532362 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.532327    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"]
Oct 30 03:01:30.532552 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.532460    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.611209 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.611124    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.611358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.611221    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.611358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.611299    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w42xl\" (UniqueName: \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-kube-api-access-w42xl\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.611466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.611410    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.611514 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.611496    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a68760-ccdd-4222-9463-862df0c25ed3-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.611549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.611531    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.611640 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.611626    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.611678 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.611672    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.611749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.611736    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.713359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.713320    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.713535 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.713385    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.713535 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.713522    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w42xl\" (UniqueName: \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-kube-api-access-w42xl\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.713641 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.713601    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.713687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.713663    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a68760-ccdd-4222-9463-862df0c25ed3-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.713741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.713707    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.713790 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.713774    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.713847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.713822    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.713906 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.713847    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.713962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.713934    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.714056 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.714031    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.714200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.714177    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.714368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.714342    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a68760-ccdd-4222-9463-862df0c25ed3-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.714494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.714378    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.716022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.715998    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.716022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.716013    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.732709 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.732686    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w42xl\" (UniqueName: \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-kube-api-access-w42xl\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.732808 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.732752    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:30.857776 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:30.857738    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:31.139352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:31.139266    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:31.140795 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:31.140762    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:01:31.140944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:31.140825    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 03:01:31.345850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:31.345821    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"]
Oct 30 03:01:31.347649 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:01:31.347621    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a68760_ccdd_4222_9463_862df0c25ed3.slice/crio-0ce9eca7621f295efdfe9149752f613df191c00ece3b7c3bd6b517728ab32f99 WatchSource:0}: Error finding container 0ce9eca7621f295efdfe9149752f613df191c00ece3b7c3bd6b517728ab32f99: Status 404 returned error can't find the container with id 0ce9eca7621f295efdfe9149752f613df191c00ece3b7c3bd6b517728ab32f99
Oct 30 03:01:31.351073 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:31.351039    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:31.351177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:31.351108    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:31.351177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:31.351137    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:32.139334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:32.139302    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:01:32.139534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:32.139353    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 03:01:32.330523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:32.330474    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" event={"ID":"b2a68760-ccdd-4222-9463-862df0c25ed3","Type":"ContainerStarted","Data":"989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339"}
Oct 30 03:01:32.330736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:32.330555    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" event={"ID":"b2a68760-ccdd-4222-9463-862df0c25ed3","Type":"ContainerStarted","Data":"0ce9eca7621f295efdfe9149752f613df191c00ece3b7c3bd6b517728ab32f99"}
Oct 30 03:01:32.414567 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:32.410591    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" podStartSLOduration=2.410570051 podStartE2EDuration="2.410570051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:32.406316576 +0000 UTC m=+1230.330524768" watchObservedRunningTime="2026-10-30 03:01:32.410570051 +0000 UTC m=+1230.334778240"
Oct 30 03:01:32.866446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:32.862763    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:32.866656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:32.866525    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 03:01:32.866656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:32.866593    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 03:01:33.139732 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:33.139655    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:01:33.139732 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:33.139714    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 03:01:33.870450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:33.867473    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 03:01:33.870450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:33.867575    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 03:01:34.139889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:34.139804    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:01:34.139889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:34.139855    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 03:01:34.862983 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:34.862942    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 03:01:34.863442 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:34.863377    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 03:01:35.140048 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:35.139966    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:01:35.140545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:35.140515    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 03:01:35.865451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:35.862759    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 03:01:35.865451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:35.862822    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 03:01:36.139524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:36.139394    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:01:36.139679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:36.139511    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 03:01:36.858147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:36.858113    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 03:01:36.858620 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:36.858169    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 03:01:37.139661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:37.139583    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:01:37.139661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:37.139640    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 03:01:37.858281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:37.858247    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 03:01:37.858774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:37.858298    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 03:01:38.567717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:38.567686    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:38.568513 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:38.568491    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:38.569074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:38.569054    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:01:39.052270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:39.052229    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:39.052956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:39.052935    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:39.055002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:39.054977    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:01:48.736555 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:48.736519    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"]
Oct 30 03:01:53.737004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:53.736925    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj" podUID="2de19764-c41a-4256-a0c4-34d729e69b3c" containerName="payload-pre-processing" containerID="cri-o://1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4" gracePeriod=25
Oct 30 03:01:54.081356 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.081320    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:01:54.194006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.193557    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de19764-c41a-4256-a0c4-34d729e69b3c-kube-api-access-dh4cs\" (UniqueName: \"kubernetes.io/projected/2de19764-c41a-4256-a0c4-34d729e69b3c-kube-api-access-dh4cs\") pod \"2de19764-c41a-4256-a0c4-34d729e69b3c\" (UID: \"2de19764-c41a-4256-a0c4-34d729e69b3c\") "
Oct 30 03:01:54.194006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.193689    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de19764-c41a-4256-a0c4-34d729e69b3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de19764-c41a-4256-a0c4-34d729e69b3c-plugins-config-volume\") pod \"2de19764-c41a-4256-a0c4-34d729e69b3c\" (UID: \"2de19764-c41a-4256-a0c4-34d729e69b3c\") "
Oct 30 03:01:54.195350 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.195303    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de19764-c41a-4256-a0c4-34d729e69b3c-plugins-config-volume" pod "2de19764-c41a-4256-a0c4-34d729e69b3c" (UID: "2de19764-c41a-4256-a0c4-34d729e69b3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:01:54.225499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.198902    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de19764-c41a-4256-a0c4-34d729e69b3c-kube-api-access-dh4cs" pod "2de19764-c41a-4256-a0c4-34d729e69b3c" (UID: "2de19764-c41a-4256-a0c4-34d729e69b3c"). InnerVolumeSpecName "kube-api-access-dh4cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:01:54.225499 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:01:54.210934    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:01:54.299536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.298276    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de19764-c41a-4256-a0c4-34d729e69b3c-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:54.299536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.298311    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh4cs\" (UniqueName: \"kubernetes.io/projected/2de19764-c41a-4256-a0c4-34d729e69b3c-kube-api-access-dh4cs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:01:54.614576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.614436    4278 generic.go:356] "Generic (PLEG): container finished" podID="2de19764-c41a-4256-a0c4-34d729e69b3c" containerID="1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4" exitCode=0
Oct 30 03:01:54.614576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.614520    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj" event={"ID":"2de19764-c41a-4256-a0c4-34d729e69b3c","Type":"ContainerDied","Data":"1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4"}
Oct 30 03:01:54.614920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.614601    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj" event={"ID":"2de19764-c41a-4256-a0c4-34d729e69b3c","Type":"ContainerDied","Data":"8181d28d8a754a9efb4a6f8da1dd2f87a67ab4139da2ec82335cc8683e5cb2a2"}
Oct 30 03:01:54.614920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.614629    4278 scope.go:122] "RemoveContainer" containerID="1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4"
Oct 30 03:01:54.614920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.614806    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"
Oct 30 03:01:54.629842 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.629799    4278 scope.go:122] "RemoveContainer" containerID="1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4"
Oct 30 03:01:54.630355 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:01:54.630326    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4\": container with ID starting with 1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4 not found: ID does not exist" containerID="1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4"
Oct 30 03:01:54.630494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.630367    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4"} err="failed to get container status \"1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4\": rpc error: code = NotFound desc = could not find container \"1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4\": container with ID starting with 1bbc226903e170df9f214553c6346508893af633251476eca0a6b590c4f143d4 not found: ID does not exist"
Oct 30 03:01:54.670843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.670805    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"]
Oct 30 03:01:54.681301 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:54.681272    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9ca6fbc2-f4894c87-kgcdj"]
Oct 30 03:01:56.202752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:01:56.202713    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de19764-c41a-4256-a0c4-34d729e69b3c" path="/var/lib/kubelet/pods/2de19764-c41a-4256-a0c4-34d729e69b3c/volumes"
Oct 30 03:02:02.476001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:02.475946    4278 scope.go:122] "RemoveContainer" containerID="ffe3aabae8697739e8dc6bb42ce95a5429476d3313d33558bd4c7b11893432be"
Oct 30 03:02:02.488523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:02.488502    4278 scope.go:122] "RemoveContainer" containerID="f08678fea201d7c9f4a5fa20bcb111e944a9b0c7e09bedefb9617f44a74ababc"
Oct 30 03:02:02.499511 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:02.499477    4278 scope.go:122] "RemoveContainer" containerID="78ff014415d76ee74def849e8651528dc58e1fb2f63a276e64d469b43456c7be"
Oct 30 03:02:02.513587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:02.513557    4278 scope.go:122] "RemoveContainer" containerID="8246cdf9ae5b39d87700a45773baf044ac34911b9c4e5ebf2032fd64288daaf6"
Oct 30 03:02:02.524461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:02.524410    4278 scope.go:122] "RemoveContainer" containerID="aad74f7c897b0a86d5dd2adcf891f97c6df675403b1fead7dab74450158081b1"
Oct 30 03:02:02.537460 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:02.537403    4278 scope.go:122] "RemoveContainer" containerID="1de55fdf1f041296bbd668fa13f26917b490e06c625d3c26b666ddfceb619e88"
Oct 30 03:02:24.233828 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:02:24.233783    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:02:28.037291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:28.037250    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"]
Oct 30 03:02:28.048560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:28.048528    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82edc560-hpvx8"]
Oct 30 03:02:28.203501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:28.203461    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c22982-e253-4d6f-ab95-1f0f15af4db0" path="/var/lib/kubelet/pods/63c22982-e253-4d6f-ab95-1f0f15af4db0/volumes"
Oct 30 03:02:30.657813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.657772    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"]
Oct 30 03:02:30.658804 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.658782    4278 cpu_manager.go:414] "containerMap: removing container" podUID="2de19764-c41a-4256-a0c4-34d729e69b3c" containerName="payload-pre-processing"
Oct 30 03:02:30.658942 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.658807    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de19764-c41a-4256-a0c4-34d729e69b3c" containerName="payload-pre-processing"
Oct 30 03:02:30.659025 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.659010    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de19764-c41a-4256-a0c4-34d729e69b3c" containerName="payload-pre-processing"
Oct 30 03:02:30.662937 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.662912    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.698796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.698758    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"]
Oct 30 03:02:30.831037 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.830996    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.831236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.831064    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.831236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.831192    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.831337 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.831244    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6037223-173b-46fb-87fd-720b538cdf49-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.831372 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.831360    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.831450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.831406    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6skkr\" (UniqueName: \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-kube-api-access-6skkr\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.831593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.831541    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.831659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.831600    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6037223-173b-46fb-87fd-720b538cdf49-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.831659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.831637    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.933582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.933455    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.933582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.933573    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6skkr\" (UniqueName: \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-kube-api-access-6skkr\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.933934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.933645    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.933934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.933698    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6037223-173b-46fb-87fd-720b538cdf49-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.933934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.933751    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.933934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.933753    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.934176 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.934082    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.934270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.934244    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.934463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.934318    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.934463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.934389    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.934463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.934418    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6037223-173b-46fb-87fd-720b538cdf49-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.934961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.934932    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6037223-173b-46fb-87fd-720b538cdf49-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.935237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.935213    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.935509 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.935486    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.936531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.936489    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.937408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.937386    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6037223-173b-46fb-87fd-720b538cdf49-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.954039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.954015    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.954126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.954092    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6skkr\" (UniqueName: \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-kube-api-access-6skkr\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:30.980030 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:30.980004    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:31.320403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:31.320364    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"]
Oct 30 03:02:31.329190 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:02:31.329157    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6037223_173b_46fb_87fd_720b538cdf49.slice/crio-749ccfcafc62b1a6b7f2a7b6c073beaabdb900b7d191d13067aa1737be25b0d4 WatchSource:0}: Error finding container 749ccfcafc62b1a6b7f2a7b6c073beaabdb900b7d191d13067aa1737be25b0d4: Status 404 returned error can't find the container with id 749ccfcafc62b1a6b7f2a7b6c073beaabdb900b7d191d13067aa1737be25b0d4
Oct 30 03:02:31.333745 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:31.333708    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:31.333875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:31.333853    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:31.333936 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:31.333897    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:31.989917 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:31.989879    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" event={"ID":"c6037223-173b-46fb-87fd-720b538cdf49","Type":"ContainerStarted","Data":"3c639f4afc4b255bb0aada0c6aca4a550e8c792fc3d49e2f45b82f34e5301f7a"}
Oct 30 03:02:31.990607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:31.989934    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" event={"ID":"c6037223-173b-46fb-87fd-720b538cdf49","Type":"ContainerStarted","Data":"749ccfcafc62b1a6b7f2a7b6c073beaabdb900b7d191d13067aa1737be25b0d4"}
Oct 30 03:02:32.046060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:32.045985    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" podStartSLOduration=2.045964051 podStartE2EDuration="2.045964051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:32.044284278 +0000 UTC m=+1289.968492467" watchObservedRunningTime="2026-10-30 03:02:32.045964051 +0000 UTC m=+1289.970172239"
Oct 30 03:02:32.982449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:32.981785    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:32.983954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:32.983918    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:02:32.984103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:32.983992    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" podUID="c6037223-173b-46fb-87fd-720b538cdf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 03:02:33.983118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:33.983079    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:02:33.983769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:33.983718    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" podUID="c6037223-173b-46fb-87fd-720b538cdf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 03:02:34.983484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:34.981863    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:02:34.983484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:34.981935    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" podUID="c6037223-173b-46fb-87fd-720b538cdf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 03:02:35.987474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:35.982594    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:02:35.987474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:35.982655    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" podUID="c6037223-173b-46fb-87fd-720b538cdf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 03:02:36.988449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:36.987819    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:02:36.988449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:36.987897    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" podUID="c6037223-173b-46fb-87fd-720b538cdf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 03:02:37.798363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:37.798328    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:02:37.983330 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:37.982862    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:02:37.983330 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:37.982925    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" podUID="c6037223-173b-46fb-87fd-720b538cdf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 03:02:39.302183 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:39.302149    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:39.302726 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:39.302539    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:39.303297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:39.303277    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:02:40.037905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:40.037865    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"]
Oct 30 03:02:40.046655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:40.046618    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9e64a7-68cd9"]
Oct 30 03:02:40.206575 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:40.204065    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2374aaf2-f128-4912-9894-8e71ff9b7c5d" path="/var/lib/kubelet/pods/2374aaf2-f128-4912-9894-8e71ff9b7c5d/volumes"
Oct 30 03:02:51.030309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.030234    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"]
Oct 30 03:02:51.040108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.040071    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d749b6-g9l65"]
Oct 30 03:02:51.387301 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.387206    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"]
Oct 30 03:02:51.392188 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.392159    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.429864 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.429835    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-ae1dd9-openshift-default-dockercfg-bvl72\""
Oct 30 03:02:51.436929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.436876    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"]
Oct 30 03:02:51.513967 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.513912    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7219839-2d46-4a53-9a67-14a21f585686-istio-podinfo\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.514140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.513979    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-credential-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.514140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.514114    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2whl\" (UniqueName: \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-kube-api-access-n2whl\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.514259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.514192    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-data\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.514324 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.514308    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-istio-token\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.514383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.514358    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.514448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.514412    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7219839-2d46-4a53-9a67-14a21f585686-istiod-ca-cert\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.514611 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.514595    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-certs\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.514746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.514727    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-envoy\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.616646    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7219839-2d46-4a53-9a67-14a21f585686-istiod-ca-cert\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.616754    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-certs\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.616844    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-envoy\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.616954    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7219839-2d46-4a53-9a67-14a21f585686-istio-podinfo\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.616995    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-credential-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.617057    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2whl\" (UniqueName: \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-kube-api-access-n2whl\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.617119    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-data\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.617233    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-istio-token\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.617286    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.617693    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.618358    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7219839-2d46-4a53-9a67-14a21f585686-istiod-ca-cert\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.618723    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-credential-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.618806    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-certs\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.619478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.618959    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-data\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.628453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.624111    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-envoy\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.628453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.624919    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7219839-2d46-4a53-9a67-14a21f585686-istio-podinfo\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.662492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.655988    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-istio-token\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.681450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.672359    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2whl\" (UniqueName: \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-kube-api-access-n2whl\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:51.720452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:51.715735    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:52.121089 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:52.121051    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"]
Oct 30 03:02:52.129779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:52.129743    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:52.129913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:52.129816    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:52.129913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:52.129844    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:52.201482 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:52.201409    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74730ba-4801-4ff7-ba08-1a080440a524" path="/var/lib/kubelet/pods/a74730ba-4801-4ff7-ba08-1a080440a524/volumes"
Oct 30 03:02:52.201825 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:52.201804    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" event={"ID":"a7219839-2d46-4a53-9a67-14a21f585686","Type":"ContainerStarted","Data":"793622c069c908ddd43cafeb6de74e107c3bd31bdd6de266315d0a46ac3df6be"}
Oct 30 03:02:53.208442 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:53.208393    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" event={"ID":"a7219839-2d46-4a53-9a67-14a21f585686","Type":"ContainerStarted","Data":"ad14d940efda53dfd9e2e444ff77a25676c72ae7325a596432a3ef643238ee7a"}
Oct 30 03:02:53.276163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:53.276089    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" podStartSLOduration=2.27606906 podStartE2EDuration="2.27606906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:53.262506566 +0000 UTC m=+1311.186714766" watchObservedRunningTime="2026-10-30 03:02:53.27606906 +0000 UTC m=+1311.200277250"
Oct 30 03:02:53.716969 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:53.716929    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:53.723095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:53.723071    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:54.191154 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:02:54.191115    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:02:54.215049 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:54.214975    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:54.216052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:54.216029    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:02:55.422147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.422113    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"]
Oct 30 03:02:55.427816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.426706    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.441362 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.441336    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9d3552-openshift-default-dockercfg-k8rl8\""
Oct 30 03:02:55.463677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.459412    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"]
Oct 30 03:02:55.485592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.485556    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-istio-token\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.485744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.485626    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-data\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.485795 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.485757    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffac99be-3709-4c4f-9526-7c290027b026-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.485949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.485931    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-credential-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.486000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.485967    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.486077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.486059    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-envoy\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.486153 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.486137    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffac99be-3709-4c4f-9526-7c290027b026-istio-podinfo\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.486188 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.486175    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92zgz\" (UniqueName: \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-kube-api-access-92zgz\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.486328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.486307    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-certs\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.587624    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-credential-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.587684    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.587725    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-envoy\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.587780    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffac99be-3709-4c4f-9526-7c290027b026-istio-podinfo\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.587811    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92zgz\" (UniqueName: \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-kube-api-access-92zgz\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.587885    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-certs\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.588039    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-istio-token\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.588120    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-data\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.588172    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffac99be-3709-4c4f-9526-7c290027b026-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.589065    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffac99be-3709-4c4f-9526-7c290027b026-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.590657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.590492    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-credential-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.591401 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.590743    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.608712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.591944    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-certs\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.608712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.593110    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-data\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.630476 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.619917    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffac99be-3709-4c4f-9526-7c290027b026-istio-podinfo\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.665445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.643384    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-istio-token\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.665445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.643778    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-envoy\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.694498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.676594    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92zgz\" (UniqueName: \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-kube-api-access-92zgz\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:55.744449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:55.743766    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:56.144412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:56.144371    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"]
Oct 30 03:02:56.154901 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:02:56.154874    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffac99be_3709_4c4f_9526_7c290027b026.slice/crio-3c5184c7c93490b8b201b812bf0de5c106c8dea0b8190160731e36560f44c0ca WatchSource:0}: Error finding container 3c5184c7c93490b8b201b812bf0de5c106c8dea0b8190160731e36560f44c0ca: Status 404 returned error can't find the container with id 3c5184c7c93490b8b201b812bf0de5c106c8dea0b8190160731e36560f44c0ca
Oct 30 03:02:56.157757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:56.157725    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:56.157854 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:56.157796    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:56.157854 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:56.157834    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:56.233958 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:56.233923    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" event={"ID":"ffac99be-3709-4c4f-9526-7c290027b026","Type":"ContainerStarted","Data":"3c5184c7c93490b8b201b812bf0de5c106c8dea0b8190160731e36560f44c0ca"}
Oct 30 03:02:57.028579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:57.028543    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"]
Oct 30 03:02:57.038677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:57.038650    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a1b293-hmf8j"]
Oct 30 03:02:57.243140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:57.243105    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" event={"ID":"ffac99be-3709-4c4f-9526-7c290027b026","Type":"ContainerStarted","Data":"17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b"}
Oct 30 03:02:57.287955 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:57.287857    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" podStartSLOduration=2.287842335 podStartE2EDuration="2.287842335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:57.282832263 +0000 UTC m=+1315.207040460" watchObservedRunningTime="2026-10-30 03:02:57.287842335 +0000 UTC m=+1315.212050523"
Oct 30 03:02:57.744297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:57.744207    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:57.750676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:57.750653    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:58.214624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:58.214578    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675d2806-7d9f-4b20-8138-943eb40781e2" path="/var/lib/kubelet/pods/675d2806-7d9f-4b20-8138-943eb40781e2/volumes"
Oct 30 03:02:58.255267 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:58.255112    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:58.256841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:58.256819    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:02:59.070279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.070244    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"]
Oct 30 03:02:59.073887 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.073863    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.100099 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.100066    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"]
Oct 30 03:02:59.159774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.159741    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.159976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.159810    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.159976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.159950    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.160109 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.160088    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.160163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.160120    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.160219 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.160167    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8xm\" (UniqueName: \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-kube-api-access-lm8xm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.160219 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.160191    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.160320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.160229    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.160320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.160314    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.261380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.261347    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.261863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.261436    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.261863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.261492    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.261863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.261599    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.261863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.261661    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.261863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.261853    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.262119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.261943    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.262119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.261998    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.262119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.262049    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.262119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.262083    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.262351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.262120    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8xm\" (UniqueName: \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-kube-api-access-lm8xm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.262351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.262233    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.262525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.262413    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.263027 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.263003    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.264138 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.264111    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.264251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.264210    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.280256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.280232    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.280338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.280290    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8xm\" (UniqueName: \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-kube-api-access-lm8xm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kdl79\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.389574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.389491    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:02:59.708402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.708371    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"]
Oct 30 03:02:59.709487 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:02:59.709462    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a4aa09_1b32_4ae5_80e3_43bc0a2ab6b7.slice/crio-82c64185cd8901c312c4cf242aa2b604db4176cc8dc4f14a6c30551f84240142 WatchSource:0}: Error finding container 82c64185cd8901c312c4cf242aa2b604db4176cc8dc4f14a6c30551f84240142: Status 404 returned error can't find the container with id 82c64185cd8901c312c4cf242aa2b604db4176cc8dc4f14a6c30551f84240142
Oct 30 03:02:59.712241 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.712205    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:59.712318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.712274    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:59.712318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:02:59.712305    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:00.273955 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:00.273913    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" event={"ID":"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7","Type":"ContainerStarted","Data":"a62b472d8f878f0394e45bc854e9e1a082369fe50011819638d6433c98b4aca3"}
Oct 30 03:03:00.274937 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:00.273980    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" event={"ID":"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7","Type":"ContainerStarted","Data":"82c64185cd8901c312c4cf242aa2b604db4176cc8dc4f14a6c30551f84240142"}
Oct 30 03:03:00.342946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:00.342897    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" podStartSLOduration=1.3428823300000001 podStartE2EDuration="1.34288233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:00.328567469 +0000 UTC m=+1318.252775657" watchObservedRunningTime="2026-10-30 03:03:00.34288233 +0000 UTC m=+1318.267090517"
Oct 30 03:03:00.389788 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:00.389744    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:03:00.391284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:00.391252    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kdl79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 03:03:00.391443 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:00.391312    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 03:03:01.390364 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:01.390323    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kdl79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 03:03:01.390820 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:01.390378    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 03:03:02.392450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:02.390285    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kdl79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 03:03:02.392450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:02.390353    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 03:03:03.390623 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:03.390585    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kdl79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 03:03:03.390826 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:03.390649    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 03:03:04.390283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:04.390247    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kdl79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 03:03:04.390829 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:04.390323    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 03:03:05.389957 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:05.389920    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kdl79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 03:03:05.390140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:05.389970    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 03:03:06.390848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:06.390803    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kdl79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 03:03:06.391278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:06.390874    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 03:03:07.390252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:07.390211    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kdl79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 03:03:07.390472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:07.390284    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 03:03:08.642286 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:08.642255    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:03:08.642744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:08.642654    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:03:08.643596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:08.643570    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:03:09.027977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:09.027940    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"]
Oct 30 03:03:09.031869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:09.031834    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc01355-pr67m"]
Oct 30 03:03:10.201662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:10.201629    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c020155-58a3-4239-acb6-190c3edee0de" path="/var/lib/kubelet/pods/7c020155-58a3-4239-acb6-190c3edee0de/volumes"
Oct 30 03:03:14.524635 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.524600    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"]
Oct 30 03:03:14.528783 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.528763    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.556554 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.556527    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d7fde0-openshift-default-dockercfg-jgzbw\""
Oct 30 03:03:14.559221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.559189    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"]
Oct 30 03:03:14.699015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.697622    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-socket\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.699015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.697884    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-certs\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.699015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.697933    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-credential-socket\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.699015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.697999    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzrw\" (UniqueName: \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-kube-api-access-xtzrw\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.699015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.698030    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istiod-ca-cert\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.699015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.698065    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-token\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.699015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.698196    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-envoy\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.699015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.698242    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-podinfo\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.699015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.698304    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-data\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.801545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.800772    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-certs\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.801545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.800833    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-credential-socket\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.801545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.800899    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzrw\" (UniqueName: \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-kube-api-access-xtzrw\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.801545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.800932    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istiod-ca-cert\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.801545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.800966    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-token\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.801545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.801083    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-envoy\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.801545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.801134    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-podinfo\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.801545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.801191    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-data\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.801545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.801260    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-socket\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.802126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.801820    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-socket\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.802126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.802071    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-certs\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.802361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.802300    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-credential-socket\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.803284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.803258    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istiod-ca-cert\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.809869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.809536    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-envoy\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.809869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.809787    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-data\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.825623 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.825206    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzrw\" (UniqueName: \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-kube-api-access-xtzrw\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.844311 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.840499    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-podinfo\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.844311 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.841902    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-token\") pod \"e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") " pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:14.846368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:14.846344    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:15.193145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:15.193115    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"]
Oct 30 03:03:15.201256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:15.201210    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:15.201357 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:15.201311    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:15.201418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:15.201357    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:15.487312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:15.487219    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" event={"ID":"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59","Type":"ContainerStarted","Data":"1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03"}
Oct 30 03:03:15.487312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:15.487296    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" event={"ID":"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59","Type":"ContainerStarted","Data":"ceccb4ee225e0a2a27138378b9e4e1000d12c9f654f9a09b437324269e2935b8"}
Oct 30 03:03:15.531332 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:15.531269    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" podStartSLOduration=1.531249563 podStartE2EDuration="1.531249563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:15.52684934 +0000 UTC m=+1333.451057559" watchObservedRunningTime="2026-10-30 03:03:15.531249563 +0000 UTC m=+1333.455457751"
Oct 30 03:03:15.847243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:15.847209    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:16.029543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:16.029506    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"]
Oct 30 03:03:16.033569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:16.033544    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee30a2a1-zhnd6"]
Oct 30 03:03:16.204598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:16.204514    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5" path="/var/lib/kubelet/pods/5cb69b6c-d31d-41ed-b9b3-cc3e1f3515a5/volumes"
Oct 30 03:03:16.858148 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:16.858113    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:17.503591 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:17.503557    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:17.504570 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:17.504550    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:03:24.192449 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:03:24.192392    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:03:31.036849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:31.036807    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"]
Oct 30 03:03:31.042707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:31.042676    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-290cb2f6-6nzfb"]
Oct 30 03:03:32.201235 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:32.201194    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe67cad2-4e90-4721-8b5d-4c91538982c6" path="/var/lib/kubelet/pods/fe67cad2-4e90-4721-8b5d-4c91538982c6/volumes"
Oct 30 03:03:40.707684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:40.707651    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:03:47.512030 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.511993    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"]
Oct 30 03:03:47.518272 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.518251    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.536539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.536509    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"]
Oct 30 03:03:47.602491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.600186    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d7fde0-serving-cert\""
Oct 30 03:03:47.713372 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.713329    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxpm\" (UniqueName: \"kubernetes.io/projected/a14e4038-6cba-41c3-9736-9a9a0e8194e5-kube-api-access-shxpm\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.714601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.713736    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-maas-api-tls\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.714601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.714161    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-metrics-certs\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.714601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.714213    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a14e4038-6cba-41c3-9736-9a9a0e8194e5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.815833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.815746    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-maas-api-tls\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.816001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.815914    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-metrics-certs\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.816001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.815944    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a14e4038-6cba-41c3-9736-9a9a0e8194e5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.816225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.816206    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxpm\" (UniqueName: \"kubernetes.io/projected/a14e4038-6cba-41c3-9736-9a9a0e8194e5-kube-api-access-shxpm\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.816749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.816721    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a14e4038-6cba-41c3-9736-9a9a0e8194e5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.818737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.818710    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-metrics-certs\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.818844 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.818792    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-maas-api-tls\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:47.836822 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:47.836791    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxpm\" (UniqueName: \"kubernetes.io/projected/a14e4038-6cba-41c3-9736-9a9a0e8194e5-kube-api-access-shxpm\") pod \"maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:48.136463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:48.136371    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:48.487870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:48.487686    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"]
Oct 30 03:03:48.493741 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:03:48.493714    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14e4038_6cba_41c3_9736_9a9a0e8194e5.slice/crio-9ac815f2be908263b6b0db70ae354484fae20da72a358e555ebb6fd68739a862 WatchSource:0}: Error finding container 9ac815f2be908263b6b0db70ae354484fae20da72a358e555ebb6fd68739a862: Status 404 returned error can't find the container with id 9ac815f2be908263b6b0db70ae354484fae20da72a358e555ebb6fd68739a862
Oct 30 03:03:48.822156 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:48.822120    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh" event={"ID":"a14e4038-6cba-41c3-9736-9a9a0e8194e5","Type":"ContainerStarted","Data":"9ac815f2be908263b6b0db70ae354484fae20da72a358e555ebb6fd68739a862"}
Oct 30 03:03:49.833798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:49.833694    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh" event={"ID":"a14e4038-6cba-41c3-9736-9a9a0e8194e5","Type":"ContainerStarted","Data":"3e239b5fe4f12f9a8b092a486ac0a7cb38e04de4448fb559a29e5549f0cd5b08"}
Oct 30 03:03:49.834164 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:49.833950    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:49.879555 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:49.879486    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh" podStartSLOduration=2.44483053 podStartE2EDuration="2.879468036s" totalImagesPullingTime="434.637506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:49.876784968 +0000 UTC m=+1367.800993156" watchObservedRunningTime="2026-10-30 03:03:49.879468036 +0000 UTC m=+1367.803676224"
Oct 30 03:03:50.474243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.474206    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"]
Oct 30 03:03:50.480189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.480166    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.507457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.507307    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"]
Oct 30 03:03:50.512004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.510063    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-be9f2a59\"/\"openshift-service-ca.crt\""
Oct 30 03:03:50.512004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.510183    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-be9f2a59\"/\"kube-root-ca.crt\""
Oct 30 03:03:50.512004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.510334    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-be9f2a59\"/\"default-dockercfg-6cj57\""
Oct 30 03:03:50.512004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.510404    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-be9f2a59\"/\"e2e-lifecycle-model-be9f2a59-kserve-self-signed-certs\""
Oct 30 03:03:50.564738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.564704    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-home\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.564900 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.564749    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tls-certs\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.565009 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.564984    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-model-cache\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.565164 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.565146    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tmp-dir\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.565222 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.565209    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-dshm\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.565290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.565276    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqb7s\" (UniqueName: \"kubernetes.io/projected/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-kube-api-access-kqb7s\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.667568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.667137    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-home\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.667568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.667185    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tls-certs\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.667568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.667307    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-model-cache\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.667568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.667454    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tmp-dir\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.667568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.667512    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-dshm\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.667568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.667576    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqb7s\" (UniqueName: \"kubernetes.io/projected/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-kube-api-access-kqb7s\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.668107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.667901    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-model-cache\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.668507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.668413    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tmp-dir\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.668637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.668537    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-home\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.670479 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.670455    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-dshm\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.671067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.671044    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tls-certs\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.695983 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.695925    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqb7s\" (UniqueName: \"kubernetes.io/projected/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-kube-api-access-kqb7s\") pod \"e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") " pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:50.814970 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:50.814688    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:51.189512 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:51.189119    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"]
Oct 30 03:03:51.888446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:51.887298    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5" event={"ID":"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e","Type":"ContainerStarted","Data":"9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4"}
Oct 30 03:03:51.888446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:51.887350    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5" event={"ID":"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e","Type":"ContainerStarted","Data":"8edb1310059b82a70ccb68ee8c0eec86b3409a95e22d1ac9af90fde3985ca6c8"}
Oct 30 03:03:52.815984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:52.815949    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:03:54.189796 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:03:54.189755    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:03:54.216765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:54.216738    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 03:03:54.216918 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:54.216793    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 03:03:54.918660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:54.918627    4278 generic.go:356] "Generic (PLEG): container finished" podID="a7219839-2d46-4a53-9a67-14a21f585686" containerID="ad14d940efda53dfd9e2e444ff77a25676c72ae7325a596432a3ef643238ee7a" exitCode=0
Oct 30 03:03:54.918660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:54.918653    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" event={"ID":"a7219839-2d46-4a53-9a67-14a21f585686","Type":"ContainerDied","Data":"ad14d940efda53dfd9e2e444ff77a25676c72ae7325a596432a3ef643238ee7a"}
Oct 30 03:03:54.919101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:54.919087    4278 scope.go:122] "RemoveContainer" containerID="ad14d940efda53dfd9e2e444ff77a25676c72ae7325a596432a3ef643238ee7a"
Oct 30 03:03:54.920887 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:54.920860    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:54.920977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:54.920928    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:54.920977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:54.920963    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:54.978723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:54.977713    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5" podStartSLOduration=4.9776962099999995 podStartE2EDuration="4.97769621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:51.975271122 +0000 UTC m=+1369.899479311" watchObservedRunningTime="2026-10-30 03:03:54.97769621 +0000 UTC m=+1372.901904399"
Oct 30 03:03:55.857637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:55.857608    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:03:55.940411 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:55.940373    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" event={"ID":"a7219839-2d46-4a53-9a67-14a21f585686","Type":"ContainerStarted","Data":"96843cdca12f093c54e2ac43f2774841fe0a97c9a1fd326b8f9401bbd47ba711"}
Oct 30 03:03:56.716727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:56.716694    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:03:56.716946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:56.716835    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 03:03:56.716946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:56.716873    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 03:03:57.717418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:57.717364    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 03:03:57.717960 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:57.717443    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 03:03:58.717466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:58.717393    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 03:03:58.717944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:58.717481    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 03:03:59.717285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:59.717240    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 03:03:59.717507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:03:59.717292    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 03:04:00.716645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:00.716607    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 03:04:00.716830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:00.716670    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 03:04:00.815799 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:00.815765    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:04:00.833780 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:00.833746    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:04:01.023569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:01.023541    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:04:01.721452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:01.721406    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:04:01.721898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:01.721880    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:04:01.722604 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:01.722584    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:04:02.626376 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:02.626342    4278 scope.go:122] "RemoveContainer" containerID="45c3ca1b08ab5b37a50f9fa081dd9f59bd07345a7bfba56781761d5618f85d4b"
Oct 30 03:04:02.636960 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:02.636943    4278 scope.go:122] "RemoveContainer" containerID="5e5f67b8e17bfead4b407c4c4e2f4b3ddfa7a5dcc8ce30b9c53c4bb768f4848e"
Oct 30 03:04:02.646129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:02.646108    4278 scope.go:122] "RemoveContainer" containerID="ef021672b5b48c6cb93d43d935a3c353aac13ce042b9334f9228c21e36a3d5e3"
Oct 30 03:04:02.655247 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:02.655218    4278 scope.go:122] "RemoveContainer" containerID="1b4e5a7b5b022f469944bf96943a898facefccff6c11d72c8556ce48736dda68"
Oct 30 03:04:03.031244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:03.031213    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"]
Oct 30 03:04:03.042395 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:03.042366    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7e3054-4qbsv"]
Oct 30 03:04:04.201119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.201086    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e85a0d-7d83-4b15-a878-276aaeaa623b" path="/var/lib/kubelet/pods/44e85a0d-7d83-4b15-a878-276aaeaa623b/volumes"
Oct 30 03:04:04.378348 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.378313    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"]
Oct 30 03:04:04.386750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.386726    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.402808 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.402785    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-44a8fd-openshift-default-dockercfg-xppxc\""
Oct 30 03:04:04.422959 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.422928    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"]
Oct 30 03:04:04.482555 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.482460    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-credential-socket\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.482833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.482801    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-podinfo\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.482976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.482866    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-socket\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.483041 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.482975    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-certs\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.483041 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.483024    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-envoy\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.483183 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.483168    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-token\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.483238 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.483225    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5c2e8d-d618-450e-81ff-04c418279c52-istiod-ca-cert\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.483327 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.483309    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-data\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.483385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.483346    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8k6s\" (UniqueName: \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-kube-api-access-j8k6s\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.585304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.585253    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-data\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.585576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.585312    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8k6s\" (UniqueName: \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-kube-api-access-j8k6s\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.585576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.585530    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-credential-socket\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.585723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.585590    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-podinfo\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.585723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.585632    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-socket\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.585723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.585692    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-certs\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.585944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.585741    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-data\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.585944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.585751    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-envoy\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.586385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.586354    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-credential-socket\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.586666 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.586644    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-certs\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.586802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.586665    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-token\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.586956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.586670    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-socket\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.587043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.586944    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5c2e8d-d618-450e-81ff-04c418279c52-istiod-ca-cert\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.587824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.587799    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5c2e8d-d618-450e-81ff-04c418279c52-istiod-ca-cert\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.589719 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.589610    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-podinfo\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.590205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.590180    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-envoy\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.604720 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.604696    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8k6s\" (UniqueName: \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-kube-api-access-j8k6s\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.605508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.605489    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-token\") pod \"e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") " pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:04.729447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:04.728689    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:05.283934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:05.283845    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"]
Oct 30 03:04:05.288459 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:04:05.288396    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5c2e8d_d618_450e_81ff_04c418279c52.slice/crio-ae5025f014973756a25046836b0d679d890768263f5b33b0dc5702e9d45106a9 WatchSource:0}: Error finding container ae5025f014973756a25046836b0d679d890768263f5b33b0dc5702e9d45106a9: Status 404 returned error can't find the container with id ae5025f014973756a25046836b0d679d890768263f5b33b0dc5702e9d45106a9
Oct 30 03:04:05.291353 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:05.291319    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:05.291490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:05.291388    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:05.291490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:05.291419    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:06.058107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:06.058072    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" event={"ID":"0e5c2e8d-d618-450e-81ff-04c418279c52","Type":"ContainerStarted","Data":"97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a"}
Oct 30 03:04:06.058308 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:06.058127    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" event={"ID":"0e5c2e8d-d618-450e-81ff-04c418279c52","Type":"ContainerStarted","Data":"ae5025f014973756a25046836b0d679d890768263f5b33b0dc5702e9d45106a9"}
Oct 30 03:04:06.114324 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:06.114251    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" podStartSLOduration=2.114235787 podStartE2EDuration="2.114235787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:06.109204972 +0000 UTC m=+1384.033413160" watchObservedRunningTime="2026-10-30 03:04:06.114235787 +0000 UTC m=+1384.038443975"
Oct 30 03:04:06.729893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:06.729861    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:06.736690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:06.736658    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:07.065896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:07.065868    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:07.066674 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:07.066656    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:04:07.640557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:07.640521    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8800-51f168"]
Oct 30 03:04:07.645051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:07.645032    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-51f168"
Oct 30 03:04:07.659607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:07.659579    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-51f168"]
Oct 30 03:04:07.665890 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:07.665864    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7q6n5\""
Oct 30 03:04:07.732949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:07.732916    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64n4\" (UniqueName: \"kubernetes.io/projected/47692d27-0c94-4598-9421-c5b70ea0b039-kube-api-access-d64n4\") pod \"test-curl-8800-51f168\" (UID: \"47692d27-0c94-4598-9421-c5b70ea0b039\") " pod="opendatahub/test-curl-8800-51f168"
Oct 30 03:04:07.834875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:07.834839    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64n4\" (UniqueName: \"kubernetes.io/projected/47692d27-0c94-4598-9421-c5b70ea0b039-kube-api-access-d64n4\") pod \"test-curl-8800-51f168\" (UID: \"47692d27-0c94-4598-9421-c5b70ea0b039\") " pod="opendatahub/test-curl-8800-51f168"
Oct 30 03:04:07.856255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:07.856221    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64n4\" (UniqueName: \"kubernetes.io/projected/47692d27-0c94-4598-9421-c5b70ea0b039-kube-api-access-d64n4\") pod \"test-curl-8800-51f168\" (UID: \"47692d27-0c94-4598-9421-c5b70ea0b039\") " pod="opendatahub/test-curl-8800-51f168"
Oct 30 03:04:07.959911 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:07.959840    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-51f168"
Oct 30 03:04:08.282480 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:08.282450    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-51f168"]
Oct 30 03:04:08.284725 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:04:08.284695    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47692d27_0c94_4598_9421_c5b70ea0b039.slice/crio-5c8a1c7ed0a81a18ff3e7217cd97f139486006c6e1f978fd2bc92f02cbea8f97 WatchSource:0}: Error finding container 5c8a1c7ed0a81a18ff3e7217cd97f139486006c6e1f978fd2bc92f02cbea8f97: Status 404 returned error can't find the container with id 5c8a1c7ed0a81a18ff3e7217cd97f139486006c6e1f978fd2bc92f02cbea8f97
Oct 30 03:04:09.084774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:09.084731    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-51f168" event={"ID":"47692d27-0c94-4598-9421-c5b70ea0b039","Type":"ContainerStarted","Data":"80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3"}
Oct 30 03:04:09.084774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:09.084781    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-51f168" event={"ID":"47692d27-0c94-4598-9421-c5b70ea0b039","Type":"ContainerStarted","Data":"5c8a1c7ed0a81a18ff3e7217cd97f139486006c6e1f978fd2bc92f02cbea8f97"}
Oct 30 03:04:09.125630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:09.125522    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8800-51f168" podStartSLOduration=1.537822204 podStartE2EDuration="2.125507585s" totalImagesPullingTime="587.685381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:09.120736897 +0000 UTC m=+1387.044945087" watchObservedRunningTime="2026-10-30 03:04:09.125507585 +0000 UTC m=+1387.049715773"
Oct 30 03:04:09.980660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:09.980621    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8800-51f168"]
Oct 30 03:04:09.985758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:09.985729    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8800-51f168"]
Oct 30 03:04:10.137122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.137089    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8800-dd8d02"]
Oct 30 03:04:10.137975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.137946    4278 cpu_manager.go:414] "containerMap: removing container" podUID="47692d27-0c94-4598-9421-c5b70ea0b039" containerName="test-curl-8800-51f168"
Oct 30 03:04:10.137975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.137971    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47692d27-0c94-4598-9421-c5b70ea0b039" containerName="test-curl-8800-51f168"
Oct 30 03:04:10.138163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.138100    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="47692d27-0c94-4598-9421-c5b70ea0b039" containerName="test-curl-8800-51f168"
Oct 30 03:04:10.141969 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.141951    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-dd8d02"
Oct 30 03:04:10.165550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.159995    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-dd8d02"]
Oct 30 03:04:10.284660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.284629    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s65xv\" (UniqueName: \"kubernetes.io/projected/65a6ce42-5721-4cb3-a310-b153c5596506-kube-api-access-s65xv\") pod \"test-curl-8800-dd8d02\" (UID: \"65a6ce42-5721-4cb3-a310-b153c5596506\") " pod="opendatahub/test-curl-8800-dd8d02"
Oct 30 03:04:10.386678 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.386650    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s65xv\" (UniqueName: \"kubernetes.io/projected/65a6ce42-5721-4cb3-a310-b153c5596506-kube-api-access-s65xv\") pod \"test-curl-8800-dd8d02\" (UID: \"65a6ce42-5721-4cb3-a310-b153c5596506\") " pod="opendatahub/test-curl-8800-dd8d02"
Oct 30 03:04:10.410533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.410503    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s65xv\" (UniqueName: \"kubernetes.io/projected/65a6ce42-5721-4cb3-a310-b153c5596506-kube-api-access-s65xv\") pod \"test-curl-8800-dd8d02\" (UID: \"65a6ce42-5721-4cb3-a310-b153c5596506\") " pod="opendatahub/test-curl-8800-dd8d02"
Oct 30 03:04:10.467991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.467961    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-dd8d02"
Oct 30 03:04:10.902115 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:10.902062    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-dd8d02"]
Oct 30 03:04:10.913970 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:04:10.913935    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65a6ce42_5721_4cb3_a310_b153c5596506.slice/crio-7f3a3502332fc50ac507335aa312ca40c5c8d9929e365e1471c1b6d3bfd8c592 WatchSource:0}: Error finding container 7f3a3502332fc50ac507335aa312ca40c5c8d9929e365e1471c1b6d3bfd8c592: Status 404 returned error can't find the container with id 7f3a3502332fc50ac507335aa312ca40c5c8d9929e365e1471c1b6d3bfd8c592
Oct 30 03:04:11.112143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:11.112095    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-dd8d02" event={"ID":"65a6ce42-5721-4cb3-a310-b153c5596506","Type":"ContainerStarted","Data":"5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54"}
Oct 30 03:04:11.112336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:11.112151    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-dd8d02" event={"ID":"65a6ce42-5721-4cb3-a310-b153c5596506","Type":"ContainerStarted","Data":"7f3a3502332fc50ac507335aa312ca40c5c8d9929e365e1471c1b6d3bfd8c592"}
Oct 30 03:04:11.112336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:11.112191    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8800-51f168" podUID="47692d27-0c94-4598-9421-c5b70ea0b039" containerName="test-curl-8800-51f168" containerID="cri-o://80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3" gracePeriod=2
Oct 30 03:04:11.149277 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:11.149216    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8800-dd8d02" podStartSLOduration=1.149196691 podStartE2EDuration="1.149196691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:11.146463713 +0000 UTC m=+1389.070671902" watchObservedRunningTime="2026-10-30 03:04:11.149196691 +0000 UTC m=+1389.073404896"
Oct 30 03:04:12.091074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.091031    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8800-dd8d02"]
Oct 30 03:04:12.099799 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.099766    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8800-dd8d02"]
Oct 30 03:04:12.258342 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.258308    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8800-46789a"]
Oct 30 03:04:12.259024 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.259007    4278 cpu_manager.go:414] "containerMap: removing container" podUID="65a6ce42-5721-4cb3-a310-b153c5596506" containerName="test-curl-8800-dd8d02"
Oct 30 03:04:12.259069 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.259025    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a6ce42-5721-4cb3-a310-b153c5596506" containerName="test-curl-8800-dd8d02"
Oct 30 03:04:12.259175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.259166    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a6ce42-5721-4cb3-a310-b153c5596506" containerName="test-curl-8800-dd8d02"
Oct 30 03:04:12.262613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.262595    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-46789a"
Oct 30 03:04:12.277635 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.277610    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-46789a"]
Oct 30 03:04:12.391563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.391398    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7vr\" (UniqueName: \"kubernetes.io/projected/1376db85-97bf-424f-93bf-5017d5d5caaa-kube-api-access-kk7vr\") pod \"test-curl-8800-46789a\" (UID: \"1376db85-97bf-424f-93bf-5017d5d5caaa\") " pod="opendatahub/test-curl-8800-46789a"
Oct 30 03:04:12.492869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.492836    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7vr\" (UniqueName: \"kubernetes.io/projected/1376db85-97bf-424f-93bf-5017d5d5caaa-kube-api-access-kk7vr\") pod \"test-curl-8800-46789a\" (UID: \"1376db85-97bf-424f-93bf-5017d5d5caaa\") " pod="opendatahub/test-curl-8800-46789a"
Oct 30 03:04:12.511092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.511061    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7vr\" (UniqueName: \"kubernetes.io/projected/1376db85-97bf-424f-93bf-5017d5d5caaa-kube-api-access-kk7vr\") pod \"test-curl-8800-46789a\" (UID: \"1376db85-97bf-424f-93bf-5017d5d5caaa\") " pod="opendatahub/test-curl-8800-46789a"
Oct 30 03:04:12.576081 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.576055    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-46789a"
Oct 30 03:04:12.875307 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:12.875279    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-46789a"]
Oct 30 03:04:12.877614 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:04:12.877557    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1376db85_97bf_424f_93bf_5017d5d5caaa.slice/crio-4a46abeaa1747986af48a8af2cab4106f5ccb0ba14f3aa3d00b519da4c988575 WatchSource:0}: Error finding container 4a46abeaa1747986af48a8af2cab4106f5ccb0ba14f3aa3d00b519da4c988575: Status 404 returned error can't find the container with id 4a46abeaa1747986af48a8af2cab4106f5ccb0ba14f3aa3d00b519da4c988575
Oct 30 03:04:13.132318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:13.132234    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-46789a" event={"ID":"1376db85-97bf-424f-93bf-5017d5d5caaa","Type":"ContainerStarted","Data":"24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6"}
Oct 30 03:04:13.132318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:13.132280    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-46789a" event={"ID":"1376db85-97bf-424f-93bf-5017d5d5caaa","Type":"ContainerStarted","Data":"4a46abeaa1747986af48a8af2cab4106f5ccb0ba14f3aa3d00b519da4c988575"}
Oct 30 03:04:13.132552 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:13.132525    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8800-dd8d02" podUID="65a6ce42-5721-4cb3-a310-b153c5596506" containerName="test-curl-8800-dd8d02" containerID="cri-o://5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54" gracePeriod=2
Oct 30 03:04:13.173718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:13.173669    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8800-46789a" podStartSLOduration=1.173654096 podStartE2EDuration="1.173654096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:13.170656433 +0000 UTC m=+1391.094864620" watchObservedRunningTime="2026-10-30 03:04:13.173654096 +0000 UTC m=+1391.097862284"
Oct 30 03:04:13.308757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:13.308732    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-51f168"
Oct 30 03:04:13.405740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:13.405659    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47692d27-0c94-4598-9421-c5b70ea0b039-kube-api-access-d64n4\" (UniqueName: \"kubernetes.io/projected/47692d27-0c94-4598-9421-c5b70ea0b039-kube-api-access-d64n4\") pod \"47692d27-0c94-4598-9421-c5b70ea0b039\" (UID: \"47692d27-0c94-4598-9421-c5b70ea0b039\") "
Oct 30 03:04:13.407736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:13.407709    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47692d27-0c94-4598-9421-c5b70ea0b039-kube-api-access-d64n4" pod "47692d27-0c94-4598-9421-c5b70ea0b039" (UID: "47692d27-0c94-4598-9421-c5b70ea0b039"). InnerVolumeSpecName "kube-api-access-d64n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:13.507644 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:13.507615    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d64n4\" (UniqueName: \"kubernetes.io/projected/47692d27-0c94-4598-9421-c5b70ea0b039-kube-api-access-d64n4\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:04:14.030090 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.030054    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8800-46789a"]
Oct 30 03:04:14.035216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.035184    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8800-46789a"]
Oct 30 03:04:14.141852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.141822    4278 generic.go:356] "Generic (PLEG): container finished" podID="47692d27-0c94-4598-9421-c5b70ea0b039" containerID="80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3" exitCode=137
Oct 30 03:04:14.142045 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.141870    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-51f168"
Oct 30 03:04:14.142045 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.141932    4278 scope.go:122] "RemoveContainer" containerID="80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3"
Oct 30 03:04:14.152225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.152211    4278 scope.go:122] "RemoveContainer" containerID="80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3"
Oct 30 03:04:14.152525 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:14.152508    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3\": container with ID starting with 80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3 not found: ID does not exist" containerID="80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3"
Oct 30 03:04:14.152586 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.152532    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3"} err="failed to get container status \"80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3\": rpc error: code = NotFound desc = could not find container \"80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3\": container with ID starting with 80aacb45717c9f9bdce77190b9434718c61eadb13e3be9faf16846255597b7e3 not found: ID does not exist"
Oct 30 03:04:14.162986 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:14.162960    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8800-51f168\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="47692d27-0c94-4598-9421-c5b70ea0b039" pod="opendatahub/test-curl-8800-51f168"
Oct 30 03:04:14.187649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.187626    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8800-fcc8cb"]
Oct 30 03:04:14.188396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.188373    4278 cpu_manager.go:414] "containerMap: removing container" podUID="1376db85-97bf-424f-93bf-5017d5d5caaa" containerName="test-curl-8800-46789a"
Oct 30 03:04:14.188396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.188393    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1376db85-97bf-424f-93bf-5017d5d5caaa" containerName="test-curl-8800-46789a"
Oct 30 03:04:14.188570 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.188534    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="1376db85-97bf-424f-93bf-5017d5d5caaa" containerName="test-curl-8800-46789a"
Oct 30 03:04:14.193562 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.193540    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-fcc8cb"
Oct 30 03:04:14.216637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.216609    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47692d27-0c94-4598-9421-c5b70ea0b039" path="/var/lib/kubelet/pods/47692d27-0c94-4598-9421-c5b70ea0b039/volumes"
Oct 30 03:04:14.216976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.216961    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-fcc8cb"]
Oct 30 03:04:14.243256 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:14.243226    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8800-51f168\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="47692d27-0c94-4598-9421-c5b70ea0b039" pod="opendatahub/test-curl-8800-51f168"
Oct 30 03:04:14.328564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.328490    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gpxr\" (UniqueName: \"kubernetes.io/projected/f1c2524f-806f-477a-8197-1a464fd7f99a-kube-api-access-5gpxr\") pod \"test-curl-8800-fcc8cb\" (UID: \"f1c2524f-806f-477a-8197-1a464fd7f99a\") " pod="opendatahub/test-curl-8800-fcc8cb"
Oct 30 03:04:14.430179 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.430148    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpxr\" (UniqueName: \"kubernetes.io/projected/f1c2524f-806f-477a-8197-1a464fd7f99a-kube-api-access-5gpxr\") pod \"test-curl-8800-fcc8cb\" (UID: \"f1c2524f-806f-477a-8197-1a464fd7f99a\") " pod="opendatahub/test-curl-8800-fcc8cb"
Oct 30 03:04:14.450082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.450047    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpxr\" (UniqueName: \"kubernetes.io/projected/f1c2524f-806f-477a-8197-1a464fd7f99a-kube-api-access-5gpxr\") pod \"test-curl-8800-fcc8cb\" (UID: \"f1c2524f-806f-477a-8197-1a464fd7f99a\") " pod="opendatahub/test-curl-8800-fcc8cb"
Oct 30 03:04:14.516050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.516022    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-fcc8cb"
Oct 30 03:04:14.820526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:14.820492    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-fcc8cb"]
Oct 30 03:04:14.821319 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:04:14.821286    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1c2524f_806f_477a_8197_1a464fd7f99a.slice/crio-76be9ecced8582a62944f86a230af1bbe55d4f2c1f5895fc4242e1e76a0177cf WatchSource:0}: Error finding container 76be9ecced8582a62944f86a230af1bbe55d4f2c1f5895fc4242e1e76a0177cf: Status 404 returned error can't find the container with id 76be9ecced8582a62944f86a230af1bbe55d4f2c1f5895fc4242e1e76a0177cf
Oct 30 03:04:15.027505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:15.027473    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"]
Oct 30 03:04:15.038351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:15.038320    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-face5b93-jhttc"]
Oct 30 03:04:15.153403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:15.153373    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-fcc8cb" event={"ID":"f1c2524f-806f-477a-8197-1a464fd7f99a","Type":"ContainerStarted","Data":"bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f"}
Oct 30 03:04:15.153589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:15.153417    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-fcc8cb" event={"ID":"f1c2524f-806f-477a-8197-1a464fd7f99a","Type":"ContainerStarted","Data":"76be9ecced8582a62944f86a230af1bbe55d4f2c1f5895fc4242e1e76a0177cf"}
Oct 30 03:04:15.153589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:15.153459    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8800-46789a" podUID="1376db85-97bf-424f-93bf-5017d5d5caaa" containerName="test-curl-8800-46789a" containerID="cri-o://24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6" gracePeriod=2
Oct 30 03:04:15.191749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:15.191705    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8800-fcc8cb" podStartSLOduration=1.191690402 podStartE2EDuration="1.191690402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:15.188068615 +0000 UTC m=+1393.112276803" watchObservedRunningTime="2026-10-30 03:04:15.191690402 +0000 UTC m=+1393.115898591"
Oct 30 03:04:15.343047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:15.343023    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-dd8d02"
Oct 30 03:04:15.448158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:15.448060    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a6ce42-5721-4cb3-a310-b153c5596506-kube-api-access-s65xv\" (UniqueName: \"kubernetes.io/projected/65a6ce42-5721-4cb3-a310-b153c5596506-kube-api-access-s65xv\") pod \"65a6ce42-5721-4cb3-a310-b153c5596506\" (UID: \"65a6ce42-5721-4cb3-a310-b153c5596506\") "
Oct 30 03:04:15.450867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:15.450834    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a6ce42-5721-4cb3-a310-b153c5596506-kube-api-access-s65xv" pod "65a6ce42-5721-4cb3-a310-b153c5596506" (UID: "65a6ce42-5721-4cb3-a310-b153c5596506"). InnerVolumeSpecName "kube-api-access-s65xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:15.552344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:15.552311    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s65xv\" (UniqueName: \"kubernetes.io/projected/65a6ce42-5721-4cb3-a310-b153c5596506-kube-api-access-s65xv\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:04:16.119760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:16.119720    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8800-fcc8cb"]
Oct 30 03:04:16.124348 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:16.124316    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8800-fcc8cb"]
Oct 30 03:04:16.163475 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:16.163405    4278 generic.go:356] "Generic (PLEG): container finished" podID="65a6ce42-5721-4cb3-a310-b153c5596506" containerID="5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54" exitCode=137
Oct 30 03:04:16.163647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:16.163502    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-dd8d02"
Oct 30 03:04:16.163647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:16.163516    4278 scope.go:122] "RemoveContainer" containerID="5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54"
Oct 30 03:04:16.174515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:16.174492    4278 scope.go:122] "RemoveContainer" containerID="5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54"
Oct 30 03:04:16.174812 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:16.174790    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54\": container with ID starting with 5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54 not found: ID does not exist" containerID="5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54"
Oct 30 03:04:16.174896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:16.174824    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54"} err="failed to get container status \"5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54\": rpc error: code = NotFound desc = could not find container \"5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54\": container with ID starting with 5292df978d1f4747a065810a35ecdd86a049427af0c7876bdaa8ac96f9004a54 not found: ID does not exist"
Oct 30 03:04:16.186950 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:16.186918    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8800-dd8d02\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="65a6ce42-5721-4cb3-a310-b153c5596506" pod="opendatahub/test-curl-8800-dd8d02"
Oct 30 03:04:16.201607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:16.201585    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a6ce42-5721-4cb3-a310-b153c5596506" path="/var/lib/kubelet/pods/65a6ce42-5721-4cb3-a310-b153c5596506/volumes"
Oct 30 03:04:16.202106 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:16.202089    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47aa8be-a438-4c31-befa-1f5a98ddffc3" path="/var/lib/kubelet/pods/d47aa8be-a438-4c31-befa-1f5a98ddffc3/volumes"
Oct 30 03:04:17.183305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.183255    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8800-fcc8cb" podUID="f1c2524f-806f-477a-8197-1a464fd7f99a" containerName="test-curl-8800-fcc8cb" containerID="cri-o://bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f" gracePeriod=2
Oct 30 03:04:17.211813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.211715    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"]
Oct 30 03:04:17.215181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.215096    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f1c2524f-806f-477a-8197-1a464fd7f99a" containerName="test-curl-8800-fcc8cb"
Oct 30 03:04:17.215347 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.215328    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c2524f-806f-477a-8197-1a464fd7f99a" containerName="test-curl-8800-fcc8cb"
Oct 30 03:04:17.215820 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.215796    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c2524f-806f-477a-8197-1a464fd7f99a" containerName="test-curl-8800-fcc8cb"
Oct 30 03:04:17.223254 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.223230    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"]
Oct 30 03:04:17.223345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.223328    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.244351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.241601    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d30e9f-openshift-default-dockercfg-kjmpv\""
Oct 30 03:04:17.321246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.316661    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-data\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.321246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.316844    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-podinfo\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.321246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.317043    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v5vv\" (UniqueName: \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-kube-api-access-8v5vv\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.321246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.317242    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-socket\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.321246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.317350    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-certs\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.321246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.317565    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-credential-socket\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.321246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.317723    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-token\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.321246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.317756    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-envoy\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.321246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.317780    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340bdf62-58b6-4dc0-9814-d3a91b88e949-istiod-ca-cert\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.419726 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.419631    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-socket\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.423566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.423535    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-certs\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.557587    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-credential-socket\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.557779    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-token\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.557818    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-envoy\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.557864    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340bdf62-58b6-4dc0-9814-d3a91b88e949-istiod-ca-cert\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.557958    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-data\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.558062    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-podinfo\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.558211    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5vv\" (UniqueName: \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-kube-api-access-8v5vv\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.421913    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-socket\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.559309    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-credential-socket\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.431308    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-certs\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.563537    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-data\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.566532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.564151    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340bdf62-58b6-4dc0-9814-d3a91b88e949-istiod-ca-cert\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.580731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.580697    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-envoy\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.599901 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.596516    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-podinfo\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.625496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.616484    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-token\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.625496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.617253    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v5vv\" (UniqueName: \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-kube-api-access-8v5vv\") pod \"e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") " pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.664755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.664723    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-46789a"
Oct 30 03:04:17.761172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.761137    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1376db85-97bf-424f-93bf-5017d5d5caaa-kube-api-access-kk7vr\" (UniqueName: \"kubernetes.io/projected/1376db85-97bf-424f-93bf-5017d5d5caaa-kube-api-access-kk7vr\") pod \"1376db85-97bf-424f-93bf-5017d5d5caaa\" (UID: \"1376db85-97bf-424f-93bf-5017d5d5caaa\") "
Oct 30 03:04:17.765213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.765179    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1376db85-97bf-424f-93bf-5017d5d5caaa-kube-api-access-kk7vr" pod "1376db85-97bf-424f-93bf-5017d5d5caaa" (UID: "1376db85-97bf-424f-93bf-5017d5d5caaa"). InnerVolumeSpecName "kube-api-access-kk7vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:17.848268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.846254    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:17.864905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:17.864867    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk7vr\" (UniqueName: \"kubernetes.io/projected/1376db85-97bf-424f-93bf-5017d5d5caaa-kube-api-access-kk7vr\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:04:18.196056 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:18.196002    4278 generic.go:356] "Generic (PLEG): container finished" podID="1376db85-97bf-424f-93bf-5017d5d5caaa" containerID="24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6" exitCode=137
Oct 30 03:04:18.196969 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:18.196205    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-46789a"
Oct 30 03:04:18.206545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:18.206523    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1376db85-97bf-424f-93bf-5017d5d5caaa" path="/var/lib/kubelet/pods/1376db85-97bf-424f-93bf-5017d5d5caaa/volumes"
Oct 30 03:04:18.207041 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:18.207027    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"]
Oct 30 03:04:18.207138 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:18.207083    4278 scope.go:122] "RemoveContainer" containerID="24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6"
Oct 30 03:04:18.214630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:18.214576    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:18.214733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:18.214679    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:18.214733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:18.214721    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:18.240179 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:18.240154    4278 scope.go:122] "RemoveContainer" containerID="24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6"
Oct 30 03:04:18.240659 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:18.240568    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6\": container with ID starting with 24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6 not found: ID does not exist" containerID="24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6"
Oct 30 03:04:18.240659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:18.240609    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6"} err="failed to get container status \"24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6\": rpc error: code = NotFound desc = could not find container \"24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6\": container with ID starting with 24a8c8cd07ac1edd4da85ba81c5c916b6ab9dcaeaedc04cb99024e50cef61fa6 not found: ID does not exist"
Oct 30 03:04:19.209283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:19.209246    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" event={"ID":"340bdf62-58b6-4dc0-9814-d3a91b88e949","Type":"ContainerStarted","Data":"296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af"}
Oct 30 03:04:19.209691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:19.209311    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" event={"ID":"340bdf62-58b6-4dc0-9814-d3a91b88e949","Type":"ContainerStarted","Data":"dee0fcc0852fc692ce6fff9950c840c95d9ce1cbed3304b94d703beebdbff031"}
Oct 30 03:04:19.267275 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:19.267218    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" podStartSLOduration=2.267194521 podStartE2EDuration="2.267194521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:19.263011067 +0000 UTC m=+1397.187219278" watchObservedRunningTime="2026-10-30 03:04:19.267194521 +0000 UTC m=+1397.191402712"
Oct 30 03:04:19.479998 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:19.479922    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-fcc8cb"
Oct 30 03:04:19.611343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:19.611312    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c2524f-806f-477a-8197-1a464fd7f99a-kube-api-access-5gpxr\" (UniqueName: \"kubernetes.io/projected/f1c2524f-806f-477a-8197-1a464fd7f99a-kube-api-access-5gpxr\") pod \"f1c2524f-806f-477a-8197-1a464fd7f99a\" (UID: \"f1c2524f-806f-477a-8197-1a464fd7f99a\") "
Oct 30 03:04:19.613526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:19.613487    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c2524f-806f-477a-8197-1a464fd7f99a-kube-api-access-5gpxr" pod "f1c2524f-806f-477a-8197-1a464fd7f99a" (UID: "f1c2524f-806f-477a-8197-1a464fd7f99a"). InnerVolumeSpecName "kube-api-access-5gpxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:19.713951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:19.713918    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gpxr\" (UniqueName: \"kubernetes.io/projected/f1c2524f-806f-477a-8197-1a464fd7f99a-kube-api-access-5gpxr\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:04:19.847079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:19.847042    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:19.853264 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:19.853240    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:20.202411 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:20.202335    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c2524f-806f-477a-8197-1a464fd7f99a" path="/var/lib/kubelet/pods/f1c2524f-806f-477a-8197-1a464fd7f99a/volumes"
Oct 30 03:04:20.217476 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:20.217441    4278 generic.go:356] "Generic (PLEG): container finished" podID="f1c2524f-806f-477a-8197-1a464fd7f99a" containerID="bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f" exitCode=137
Oct 30 03:04:20.217885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:20.217493    4278 scope.go:122] "RemoveContainer" containerID="bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f"
Oct 30 03:04:20.217885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:20.217516    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-fcc8cb"
Oct 30 03:04:20.218104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:20.218081    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:20.219043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:20.219023    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:04:20.229262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:20.229085    4278 scope.go:122] "RemoveContainer" containerID="bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f"
Oct 30 03:04:20.229415 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:20.229394    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f\": container with ID starting with bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f not found: ID does not exist" containerID="bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f"
Oct 30 03:04:20.229497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:20.229438    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f"} err="failed to get container status \"bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f\": rpc error: code = NotFound desc = could not find container \"bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f\": container with ID starting with bc86195369506966bdd1c741d240a3f8d7c5ba1c898d2d61588a93fe5676535f not found: ID does not exist"
Oct 30 03:04:22.687561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:22.687523    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:04:23.219715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.219675    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"]
Oct 30 03:04:23.224674 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.224650    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.239131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.239079    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7ffb1886-serving-cert\""
Oct 30 03:04:23.259494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.257457    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"]
Oct 30 03:04:23.276989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.276955    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-openshift-service-ca\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.277158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.277139    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-maas-api-tls\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.277349 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.277321    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz85r\" (UniqueName: \"kubernetes.io/projected/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-kube-api-access-lz85r\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.277498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.277479    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-metrics-certs\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.379255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.379219    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-metrics-certs\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.379488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.379369    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-openshift-service-ca\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.379488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.379476    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-maas-api-tls\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.379688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.379553    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz85r\" (UniqueName: \"kubernetes.io/projected/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-kube-api-access-lz85r\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.380219 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.380193    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-openshift-service-ca\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.382116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.382089    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-metrics-certs\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.382212 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.382135    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-maas-api-tls\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.403603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.403577    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz85r\" (UniqueName: \"kubernetes.io/projected/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-kube-api-access-lz85r\") pod \"maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.539483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.539400    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:23.861868 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:23.861841    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"]
Oct 30 03:04:23.863350 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:04:23.863319    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acc2e64_eb9a_411f_8cb4_e169b4edf58d.slice/crio-dae13785b41299812d938e554f923d1c4a5d74658912a2955f3925b56c5206a5 WatchSource:0}: Error finding container dae13785b41299812d938e554f923d1c4a5d74658912a2955f3925b56c5206a5: Status 404 returned error can't find the container with id dae13785b41299812d938e554f923d1c4a5d74658912a2955f3925b56c5206a5
Oct 30 03:04:24.233960 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:24.233757    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:04:24.256807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:24.256764    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs" event={"ID":"0acc2e64-eb9a-411f-8cb4-e169b4edf58d","Type":"ContainerStarted","Data":"dae13785b41299812d938e554f923d1c4a5d74658912a2955f3925b56c5206a5"}
Oct 30 03:04:25.273907 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:25.273855    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs" event={"ID":"0acc2e64-eb9a-411f-8cb4-e169b4edf58d","Type":"ContainerStarted","Data":"074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340"}
Oct 30 03:04:25.274475 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:25.273970    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:25.315632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:25.315572    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs" podStartSLOduration=1.97459584 podStartE2EDuration="2.315556962s" totalImagesPullingTime="340.961122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:25.311350363 +0000 UTC m=+1403.235558550" watchObservedRunningTime="2026-10-30 03:04:25.315556962 +0000 UTC m=+1403.239765140"
Oct 30 03:04:31.542682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:31.508597    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:04:33.547929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:33.547578    4278 patch_prober.go:29] interesting pod/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:04:33.547929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:33.547645    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:04:34.437106 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.437069    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"]
Oct 30 03:04:34.464966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.464902    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"]
Oct 30 03:04:34.465162 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.465009    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.482166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.482136    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-8e2727-serving-cert\""
Oct 30 03:04:34.589863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.589828    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.590297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.589899    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.590297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.590045    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phnkw\" (UniqueName: \"kubernetes.io/projected/42127ef5-f8a8-4c9d-9002-034162b8bdf4-kube-api-access-phnkw\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.590297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.590087    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42127ef5-f8a8-4c9d-9002-034162b8bdf4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.692151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.692045    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.692151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.692135    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.692394 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.692212    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phnkw\" (UniqueName: \"kubernetes.io/projected/42127ef5-f8a8-4c9d-9002-034162b8bdf4-kube-api-access-phnkw\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.692394 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.692245    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42127ef5-f8a8-4c9d-9002-034162b8bdf4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.699987 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:34.699952    4278 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-serving-cert: secret "maas-api-e2e-shared-b-w3-8e2727-serving-cert" not found
Oct 30 03:04:34.700201 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:34.700048    4278 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-maas-api-tls podName:42127ef5-f8a8-4c9d-9002-034162b8bdf4 nodeName:}" failed. No retries permitted until 2026-10-30 03:04:35.200027122 +0000 UTC m=+1413.124235301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-maas-api-tls") pod "maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn" (UID: "42127ef5-f8a8-4c9d-9002-034162b8bdf4") : secret "maas-api-e2e-shared-b-w3-8e2727-serving-cert" not found
Oct 30 03:04:34.700279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.693414    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42127ef5-f8a8-4c9d-9002-034162b8bdf4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.716369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.716337    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:34.716641 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:34.716619    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phnkw\" (UniqueName: \"kubernetes.io/projected/42127ef5-f8a8-4c9d-9002-034162b8bdf4-kube-api-access-phnkw\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:35.201002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:35.200957    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:35.203561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:35.203533    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:35.407352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:35.407320    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:35.774173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:35.773941    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"]
Oct 30 03:04:35.867205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:35.867169    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn" event={"ID":"42127ef5-f8a8-4c9d-9002-034162b8bdf4","Type":"ContainerStarted","Data":"efbe3287206e22af15b69d7ae36f29f48d1900088bfdffce4e5b5d77a4b28320"}
Oct 30 03:04:36.891985 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.891889    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn" event={"ID":"42127ef5-f8a8-4c9d-9002-034162b8bdf4","Type":"ContainerStarted","Data":"ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0"}
Oct 30 03:04:36.892450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.891989    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:36.893656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.893633    4278 generic.go:356] "Generic (PLEG): container finished" podID="ffac99be-3709-4c4f-9526-7c290027b026" containerID="17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b" exitCode=0
Oct 30 03:04:36.893759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.893702    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" event={"ID":"ffac99be-3709-4c4f-9526-7c290027b026","Type":"ContainerDied","Data":"17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b"}
Oct 30 03:04:36.894142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.894120    4278 scope.go:122] "RemoveContainer" containerID="17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b"
Oct 30 03:04:36.895699 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.895674    4278 generic.go:356] "Generic (PLEG): container finished" podID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerID="1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03" exitCode=0
Oct 30 03:04:36.895831 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.895816    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" event={"ID":"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59","Type":"ContainerDied","Data":"1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03"}
Oct 30 03:04:36.896217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.896200    4278 scope.go:122] "RemoveContainer" containerID="1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03"
Oct 30 03:04:36.896257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.896226    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:36.896320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.896304    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:36.896365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.896351    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:36.897954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.897927    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:36.898051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.898001    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:36.898114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.898048    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:04:36.933409 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.933355    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn" podStartSLOduration=2.302617912 podStartE2EDuration="2.933339244s" totalImagesPullingTime="630.721332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:36.929975901 +0000 UTC m=+1414.854184100" watchObservedRunningTime="2026-10-30 03:04:36.933339244 +0000 UTC m=+1414.857547433"
Oct 30 03:04:36.958814 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.958782    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"]
Oct 30 03:04:36.963190 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.963169    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:04:36.978918 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.978895    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c1abc2\""
Oct 30 03:04:36.982229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:36.982205    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"]
Oct 30 03:04:37.016634 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.016588    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c1abc2-dockercfg-d5z5k\""
Oct 30 03:04:37.047774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.047743    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a0a0b1-f21f-4c43-a509-664da02486a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896\" (UID: \"d2a0a0b1-f21f-4c43-a509-664da02486a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:04:37.048109 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.047965    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b854r\" (UniqueName: \"kubernetes.io/projected/d2a0a0b1-f21f-4c43-a509-664da02486a7-kube-api-access-b854r\") pod \"payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896\" (UID: \"d2a0a0b1-f21f-4c43-a509-664da02486a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:04:37.156515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.156394    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a0a0b1-f21f-4c43-a509-664da02486a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896\" (UID: \"d2a0a0b1-f21f-4c43-a509-664da02486a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:04:37.156689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.156553    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b854r\" (UniqueName: \"kubernetes.io/projected/d2a0a0b1-f21f-4c43-a509-664da02486a7-kube-api-access-b854r\") pod \"payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896\" (UID: \"d2a0a0b1-f21f-4c43-a509-664da02486a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:04:37.160500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.159551    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a0a0b1-f21f-4c43-a509-664da02486a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896\" (UID: \"d2a0a0b1-f21f-4c43-a509-664da02486a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:04:37.217033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.210244    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b854r\" (UniqueName: \"kubernetes.io/projected/d2a0a0b1-f21f-4c43-a509-664da02486a7-kube-api-access-b854r\") pod \"payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896\" (UID: \"d2a0a0b1-f21f-4c43-a509-664da02486a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:04:37.324448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.321823    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:04:37.705743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.705671    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"]
Oct 30 03:04:37.706142 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:04:37.706114    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a0a0b1_f21f_4c43_a509_664da02486a7.slice/crio-19e2d0b3ca20af1683bc7c337bee4455c32007a9182f59ae5b51dd281980a509 WatchSource:0}: Error finding container 19e2d0b3ca20af1683bc7c337bee4455c32007a9182f59ae5b51dd281980a509: Status 404 returned error can't find the container with id 19e2d0b3ca20af1683bc7c337bee4455c32007a9182f59ae5b51dd281980a509
Oct 30 03:04:37.907371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.907339    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" event={"ID":"ffac99be-3709-4c4f-9526-7c290027b026","Type":"ContainerStarted","Data":"86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60"}
Oct 30 03:04:37.909403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.909373    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896" event={"ID":"d2a0a0b1-f21f-4c43-a509-664da02486a7","Type":"ContainerStarted","Data":"8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5"}
Oct 30 03:04:37.909539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.909450    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896" event={"ID":"d2a0a0b1-f21f-4c43-a509-664da02486a7","Type":"ContainerStarted","Data":"19e2d0b3ca20af1683bc7c337bee4455c32007a9182f59ae5b51dd281980a509"}
Oct 30 03:04:37.909539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.909501    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:04:37.911952 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:37.911915    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" event={"ID":"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59","Type":"ContainerStarted","Data":"6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4"}
Oct 30 03:04:38.029229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:38.029174    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896" podStartSLOduration=2.029160684 podStartE2EDuration="2.029160684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:38.028160926 +0000 UTC m=+1415.952369115" watchObservedRunningTime="2026-10-30 03:04:38.029160684 +0000 UTC m=+1415.953368872"
Oct 30 03:04:38.744278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:38.744241    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:04:38.745492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:38.745460    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 03:04:38.745620 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:38.745523    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 03:04:38.872455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:38.850013    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:04:38.872455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:38.852601    4278 patch_prober.go:29] interesting pod/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:04:38.872455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:38.852671    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 03:04:39.744706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:39.744674    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 03:04:39.745097 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:39.744722    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 03:04:39.847787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:39.847754    4278 patch_prober.go:29] interesting pod/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:04:39.847962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:39.847806    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 03:04:40.744953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:40.744923    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 03:04:40.745381 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:40.745356    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 03:04:40.847079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:40.847044    4278 patch_prober.go:29] interesting pod/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:04:40.847243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:40.847096    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 03:04:41.744578 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:41.744540    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 03:04:41.744762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:41.744604    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 03:04:41.847800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:41.847762    4278 patch_prober.go:29] interesting pod/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:04:41.848180 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:41.847820    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 03:04:42.745070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:42.745033    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 03:04:42.745370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:42.745337    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 03:04:42.847214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:42.847178    4278 patch_prober.go:29] interesting pod/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:04:42.847385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:42.847237    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 03:04:42.918409 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:42.918382    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:04:43.255530 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:43.255494    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:04:43.758380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:43.758337    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:04:43.851962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:43.851928    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:04:43.852383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:43.852358    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:04:43.853135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:43.853116    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:04:44.035949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:44.035864    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:04:48.226180 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:48.226142    4278 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","besteffort","pod1376db85-97bf-424f-93bf-5017d5d5caaa"] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod1376db85-97bf-424f-93bf-5017d5d5caaa] : Timed out while waiting for systemd to remove kubepods-besteffort-pod1376db85_97bf_424f_93bf_5017d5d5caaa.slice"
Oct 30 03:04:48.226591 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:48.226183    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods besteffort pod1376db85-97bf-424f-93bf-5017d5d5caaa] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod1376db85-97bf-424f-93bf-5017d5d5caaa] : Timed out while waiting for systemd to remove kubepods-besteffort-pod1376db85_97bf_424f_93bf_5017d5d5caaa.slice" pod="opendatahub/test-curl-8800-46789a" podUID="1376db85-97bf-424f-93bf-5017d5d5caaa"
Oct 30 03:04:48.953127 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:48.953097    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:04:49.073268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:04:49.073238    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-46789a"
Oct 30 03:04:49.086283 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:49.086249    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8800-46789a\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="1376db85-97bf-424f-93bf-5017d5d5caaa" pod="opendatahub/test-curl-8800-46789a"
Oct 30 03:04:49.099128 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:49.099100    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8800-46789a\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="1376db85-97bf-424f-93bf-5017d5d5caaa" pod="opendatahub/test-curl-8800-46789a"
Oct 30 03:04:54.198972 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:04:54.198917    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:05:02.701991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.701958    4278 scope.go:122] "RemoveContainer" containerID="b4397bc185bb8ebae4e92ad54360531c0d59dbe4adcd1df312f9ec37f8e098df"
Oct 30 03:05:02.712525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.712501    4278 scope.go:122] "RemoveContainer" containerID="1fab25380a8bbebeca990f3f0f568b33f863e76f52cd0c31b8cc79f3845a1ca3"
Oct 30 03:05:02.721869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.721849    4278 scope.go:122] "RemoveContainer" containerID="ee2b0e4e0ebec593bf9c069d63399df8a19f7d5d1b54cef991f6ad57c7523024"
Oct 30 03:05:02.734088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.734062    4278 scope.go:122] "RemoveContainer" containerID="f092b5ff7dfbb91b1c5223c6da9b0bc338b7bf0e5f8266e25d000ca42aff452b"
Oct 30 03:05:02.844728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.844685    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"]
Oct 30 03:05:02.860585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.860549    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:02.876818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.876569    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"]
Oct 30 03:05:02.887271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.887246    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:05:02.887483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.887322    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:05:02.935408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.935369    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcn4m\" (UniqueName: \"kubernetes.io/projected/e49bddf5-3955-4a4d-ad6a-a432033350ee-kube-api-access-zcn4m\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:02.935636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.935533    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:02.935636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:02.935613    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:03.036913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:03.036874    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcn4m\" (UniqueName: \"kubernetes.io/projected/e49bddf5-3955-4a4d-ad6a-a432033350ee-kube-api-access-zcn4m\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:03.037113 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:03.037016    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:03.037113 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:03.037093    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:03.037844 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:03.037818    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:03.039752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:03.039731    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:03.058946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:03.058918    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcn4m\" (UniqueName: \"kubernetes.io/projected/e49bddf5-3955-4a4d-ad6a-a432033350ee-kube-api-access-zcn4m\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:03.185017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:03.184983    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:03.496708 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:03.496678    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"]
Oct 30 03:05:03.499220 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:05:03.499194    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49bddf5_3955_4a4d_ad6a_a432033350ee.slice/crio-0779ce706a542055a356a3f846aa6b05900523e7f7ea4557ee4394f046472fe5 WatchSource:0}: Error finding container 0779ce706a542055a356a3f846aa6b05900523e7f7ea4557ee4394f046472fe5: Status 404 returned error can't find the container with id 0779ce706a542055a356a3f846aa6b05900523e7f7ea4557ee4394f046472fe5
Oct 30 03:05:04.225521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:04.220287    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj" event={"ID":"e49bddf5-3955-4a4d-ad6a-a432033350ee","Type":"ContainerStarted","Data":"7597080b1f5975e6363bba9f5c3782142a61c936b5583adf3d13d5d7ce41f50a"}
Oct 30 03:05:04.225521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:04.220364    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj" event={"ID":"e49bddf5-3955-4a4d-ad6a-a432033350ee","Type":"ContainerStarted","Data":"0779ce706a542055a356a3f846aa6b05900523e7f7ea4557ee4394f046472fe5"}
Oct 30 03:05:04.280447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:04.279060    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj" podStartSLOduration=2.279036252 podStartE2EDuration="2.279036252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:04.263619555 +0000 UTC m=+1442.187827744" watchObservedRunningTime="2026-10-30 03:05:04.279036252 +0000 UTC m=+1442.203244442"
Oct 30 03:05:05.229629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:05.229596    4278 generic.go:356] "Generic (PLEG): container finished" podID="e49bddf5-3955-4a4d-ad6a-a432033350ee" containerID="7597080b1f5975e6363bba9f5c3782142a61c936b5583adf3d13d5d7ce41f50a" exitCode=0
Oct 30 03:05:05.230098 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:05.229679    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj" event={"ID":"e49bddf5-3955-4a4d-ad6a-a432033350ee","Type":"ContainerDied","Data":"7597080b1f5975e6363bba9f5c3782142a61c936b5583adf3d13d5d7ce41f50a"}
Oct 30 03:05:06.546277 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:06.546243    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:06.570077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:06.570038    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-xks-ca\") pod \"e49bddf5-3955-4a4d-ad6a-a432033350ee\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") "
Oct 30 03:05:06.570366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:06.570346    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-openshift-ca\") pod \"e49bddf5-3955-4a4d-ad6a-a432033350ee\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") "
Oct 30 03:05:06.570481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:06.570456    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49bddf5-3955-4a4d-ad6a-a432033350ee-kube-api-access-zcn4m\" (UniqueName: \"kubernetes.io/projected/e49bddf5-3955-4a4d-ad6a-a432033350ee-kube-api-access-zcn4m\") pod \"e49bddf5-3955-4a4d-ad6a-a432033350ee\" (UID: \"e49bddf5-3955-4a4d-ad6a-a432033350ee\") "
Oct 30 03:05:06.578390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:06.577933    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-openshift-ca" pod "e49bddf5-3955-4a4d-ad6a-a432033350ee" (UID: "e49bddf5-3955-4a4d-ad6a-a432033350ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:06.590671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:06.590625    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-xks-ca" pod "e49bddf5-3955-4a4d-ad6a-a432033350ee" (UID: "e49bddf5-3955-4a4d-ad6a-a432033350ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:05:06.592173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:06.592130    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49bddf5-3955-4a4d-ad6a-a432033350ee-kube-api-access-zcn4m" pod "e49bddf5-3955-4a4d-ad6a-a432033350ee" (UID: "e49bddf5-3955-4a4d-ad6a-a432033350ee"). InnerVolumeSpecName "kube-api-access-zcn4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:06.690824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:06.690784    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:05:06.690824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:06.690825    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcn4m\" (UniqueName: \"kubernetes.io/projected/e49bddf5-3955-4a4d-ad6a-a432033350ee-kube-api-access-zcn4m\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:05:06.691031 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:06.690839    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49bddf5-3955-4a4d-ad6a-a432033350ee-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:05:07.245996 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:07.245953    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj" event={"ID":"e49bddf5-3955-4a4d-ad6a-a432033350ee","Type":"ContainerDied","Data":"0779ce706a542055a356a3f846aa6b05900523e7f7ea4557ee4394f046472fe5"}
Oct 30 03:05:07.245996 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:07.245983    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"
Oct 30 03:05:07.246215 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:07.246020    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0779ce706a542055a356a3f846aa6b05900523e7f7ea4557ee4394f046472fe5"
Oct 30 03:05:23.045992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.045954    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"]
Oct 30 03:05:23.053721 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.053681    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0afc2e46-x8v9x"]
Oct 30 03:05:23.821290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.821254    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"]
Oct 30 03:05:23.822067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.822049    4278 cpu_manager.go:414] "containerMap: removing container" podUID="e49bddf5-3955-4a4d-ad6a-a432033350ee" containerName="revoke-keys"
Oct 30 03:05:23.822122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.822071    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49bddf5-3955-4a4d-ad6a-a432033350ee" containerName="revoke-keys"
Oct 30 03:05:23.822217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.822200    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49bddf5-3955-4a4d-ad6a-a432033350ee" containerName="revoke-keys"
Oct 30 03:05:23.824874 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.824859    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:23.839398 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.839365    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:05:23.846151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.846127    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"]
Oct 30 03:05:23.851645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.851615    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:05:23.908375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.908316    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxfrb\" (UniqueName: \"kubernetes.io/projected/45fbac86-8232-44c0-b41b-99bfdf0ab434-kube-api-access-nxfrb\") pod \"maas-api-revoke-sub-d2010e09-jbfz9\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:23.908612 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.908592    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2010e09-jbfz9\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:23.908772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:23.908750    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2010e09-jbfz9\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:24.010096 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.010058    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2010e09-jbfz9\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:24.010285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.010257    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxfrb\" (UniqueName: \"kubernetes.io/projected/45fbac86-8232-44c0-b41b-99bfdf0ab434-kube-api-access-nxfrb\") pod \"maas-api-revoke-sub-d2010e09-jbfz9\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:24.010500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.010477    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2010e09-jbfz9\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:24.011047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.011028    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2010e09-jbfz9\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:24.012800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.012776    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2010e09-jbfz9\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:24.030438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.030396    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxfrb\" (UniqueName: \"kubernetes.io/projected/45fbac86-8232-44c0-b41b-99bfdf0ab434-kube-api-access-nxfrb\") pod \"maas-api-revoke-sub-d2010e09-jbfz9\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:24.037492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.036931    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"]
Oct 30 03:05:24.041475 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.041449    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7783122-jxgcg"]
Oct 30 03:05:24.138161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.138079    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:24.200697 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:05:24.200645    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:05:24.210306 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.210275    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c7e2c7-06ab-4253-ac03-e6d21c655dd3" path="/var/lib/kubelet/pods/94c7e2c7-06ab-4253-ac03-e6d21c655dd3/volumes"
Oct 30 03:05:24.210966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.210942    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3617e5-ed3c-41fc-bc1e-f164a1b2f937" path="/var/lib/kubelet/pods/ab3617e5-ed3c-41fc-bc1e-f164a1b2f937/volumes"
Oct 30 03:05:24.470605 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:24.470440    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"]
Oct 30 03:05:25.413159 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:25.413126    4278 generic.go:356] "Generic (PLEG): container finished" podID="45fbac86-8232-44c0-b41b-99bfdf0ab434" containerID="542ac6c44ef1fd890d4c9d2fd9dc42ee2a54a674f0e668b57ea565ed9df295cb" exitCode=0
Oct 30 03:05:25.413568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:25.413267    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9" event={"ID":"45fbac86-8232-44c0-b41b-99bfdf0ab434","Type":"ContainerDied","Data":"542ac6c44ef1fd890d4c9d2fd9dc42ee2a54a674f0e668b57ea565ed9df295cb"}
Oct 30 03:05:25.413568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:25.413342    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9" event={"ID":"45fbac86-8232-44c0-b41b-99bfdf0ab434","Type":"ContainerStarted","Data":"2458545198bcd8ec5623d45173d5e644891484037323623f07c1675dbcfbbe54"}
Oct 30 03:05:26.652675 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:26.652646    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:26.767746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:26.767710    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-openshift-ca\") pod \"45fbac86-8232-44c0-b41b-99bfdf0ab434\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") "
Oct 30 03:05:26.767971 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:26.767906    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45fbac86-8232-44c0-b41b-99bfdf0ab434-kube-api-access-nxfrb\" (UniqueName: \"kubernetes.io/projected/45fbac86-8232-44c0-b41b-99bfdf0ab434-kube-api-access-nxfrb\") pod \"45fbac86-8232-44c0-b41b-99bfdf0ab434\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") "
Oct 30 03:05:26.768039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:26.768012    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-xks-ca\") pod \"45fbac86-8232-44c0-b41b-99bfdf0ab434\" (UID: \"45fbac86-8232-44c0-b41b-99bfdf0ab434\") "
Oct 30 03:05:26.768226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:26.768197    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-openshift-ca" pod "45fbac86-8232-44c0-b41b-99bfdf0ab434" (UID: "45fbac86-8232-44c0-b41b-99bfdf0ab434"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:26.768900 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:26.768871    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:05:26.770212 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:26.770191    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fbac86-8232-44c0-b41b-99bfdf0ab434-kube-api-access-nxfrb" pod "45fbac86-8232-44c0-b41b-99bfdf0ab434" (UID: "45fbac86-8232-44c0-b41b-99bfdf0ab434"). InnerVolumeSpecName "kube-api-access-nxfrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:26.770316 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:26.770302    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-xks-ca" pod "45fbac86-8232-44c0-b41b-99bfdf0ab434" (UID: "45fbac86-8232-44c0-b41b-99bfdf0ab434"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:05:26.870689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:26.870655    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxfrb\" (UniqueName: \"kubernetes.io/projected/45fbac86-8232-44c0-b41b-99bfdf0ab434-kube-api-access-nxfrb\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:05:26.870689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:26.870685    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45fbac86-8232-44c0-b41b-99bfdf0ab434-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:05:27.432589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:27.432551    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9" event={"ID":"45fbac86-8232-44c0-b41b-99bfdf0ab434","Type":"ContainerDied","Data":"2458545198bcd8ec5623d45173d5e644891484037323623f07c1675dbcfbbe54"}
Oct 30 03:05:27.432589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:27.432581    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"
Oct 30 03:05:27.432799 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:27.432613    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2458545198bcd8ec5623d45173d5e644891484037323623f07c1675dbcfbbe54"
Oct 30 03:05:32.100564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:32.098297    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"]
Oct 30 03:05:32.100564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:32.098360    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e6d2f3-zz6wf"]
Oct 30 03:05:32.258806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:32.258765    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0843f8-3f48-4920-a716-3043351d1ea4" path="/var/lib/kubelet/pods/9a0843f8-3f48-4920-a716-3043351d1ea4/volumes"
Oct 30 03:05:37.854827 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:37.854792    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:05:38.801680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.801646    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"]
Oct 30 03:05:38.802392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.802377    4278 cpu_manager.go:414] "containerMap: removing container" podUID="45fbac86-8232-44c0-b41b-99bfdf0ab434" containerName="revoke-keys"
Oct 30 03:05:38.802486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.802394    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45fbac86-8232-44c0-b41b-99bfdf0ab434" containerName="revoke-keys"
Oct 30 03:05:38.802568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.802557    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="45fbac86-8232-44c0-b41b-99bfdf0ab434" containerName="revoke-keys"
Oct 30 03:05:38.806131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.806111    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:38.830693 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.830667    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:05:38.830880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.830667    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:05:38.835530 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.835503    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"]
Oct 30 03:05:38.902594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.902554    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1671e27-892tn\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:38.903079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.902683    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbt88\" (UniqueName: \"kubernetes.io/projected/4af4ef31-b040-46f4-82a5-709fb191bf31-kube-api-access-kbt88\") pod \"maas-api-revoke-sub-a1671e27-892tn\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:38.903079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:38.902819    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1671e27-892tn\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:39.004012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:39.003977    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1671e27-892tn\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:39.004200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:39.004124    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbt88\" (UniqueName: \"kubernetes.io/projected/4af4ef31-b040-46f4-82a5-709fb191bf31-kube-api-access-kbt88\") pod \"maas-api-revoke-sub-a1671e27-892tn\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:39.004200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:39.004172    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1671e27-892tn\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:39.004849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:39.004826    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1671e27-892tn\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:39.006354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:39.006330    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1671e27-892tn\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:39.022032 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:39.022006    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbt88\" (UniqueName: \"kubernetes.io/projected/4af4ef31-b040-46f4-82a5-709fb191bf31-kube-api-access-kbt88\") pod \"maas-api-revoke-sub-a1671e27-892tn\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:39.121673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:39.121599    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:39.632119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:39.632088    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"]
Oct 30 03:05:39.633702 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:05:39.633673    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af4ef31_b040_46f4_82a5_709fb191bf31.slice/crio-b113e00063b002524adf5fee2c2ed401ad97c605895e101bf99ab826eca5aa94 WatchSource:0}: Error finding container b113e00063b002524adf5fee2c2ed401ad97c605895e101bf99ab826eca5aa94: Status 404 returned error can't find the container with id b113e00063b002524adf5fee2c2ed401ad97c605895e101bf99ab826eca5aa94
Oct 30 03:05:40.595310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:40.595268    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn" event={"ID":"4af4ef31-b040-46f4-82a5-709fb191bf31","Type":"ContainerStarted","Data":"5146a2b93925e7ae8e3916f9f21a1fb74363543452e2e4b5d5bec06e3f094587"}
Oct 30 03:05:40.595816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:40.595339    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn" event={"ID":"4af4ef31-b040-46f4-82a5-709fb191bf31","Type":"ContainerStarted","Data":"b113e00063b002524adf5fee2c2ed401ad97c605895e101bf99ab826eca5aa94"}
Oct 30 03:05:40.638454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:40.638347    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn" podStartSLOduration=2.638323116 podStartE2EDuration="2.638323116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:40.630361708 +0000 UTC m=+1478.554569909" watchObservedRunningTime="2026-10-30 03:05:40.638323116 +0000 UTC m=+1478.562531360"
Oct 30 03:05:41.041507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:41.041466    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"]
Oct 30 03:05:41.051241 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:41.050551    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9305c8f8-fj5vr"]
Oct 30 03:05:41.671453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:41.658117    4278 generic.go:356] "Generic (PLEG): container finished" podID="4af4ef31-b040-46f4-82a5-709fb191bf31" containerID="5146a2b93925e7ae8e3916f9f21a1fb74363543452e2e4b5d5bec06e3f094587" exitCode=0
Oct 30 03:05:41.671453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:41.658174    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn" event={"ID":"4af4ef31-b040-46f4-82a5-709fb191bf31","Type":"ContainerDied","Data":"5146a2b93925e7ae8e3916f9f21a1fb74363543452e2e4b5d5bec06e3f094587"}
Oct 30 03:05:42.271454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:42.271392    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="656f02c8-4d75-42e4-8600-0861f87e2828" path="/var/lib/kubelet/pods/656f02c8-4d75-42e4-8600-0861f87e2828/volumes"
Oct 30 03:05:42.318991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:42.318946    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:42.408828 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:42.408776    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:42.450709 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:42.449562    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:42.451030 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:42.451001    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:42.480590 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:42.480548    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:42.542974 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:42.542878    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:42.543227 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:42.543189    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 30 03:05:42.571563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:42.571523    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.028076 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.028030    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5" podUID="cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.149074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.132577    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" podUID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.154466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.151827    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.237460 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.215179    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.327364 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.327267    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.341393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.341351    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.342419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.342393    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 03:05:43.471523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.471471    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.659554 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.633159    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 03:05:43.734453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.690158    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 30 03:05:43.734453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.691628    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 03:05:43.830780 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.830727    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 30 03:05:43.835287 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.835259    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 03:05:43.884784 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:43.884740    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 30 03:05:44.197550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:44.167522    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" podUID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": context deadline exceeded"
Oct 30 03:05:44.197550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:44.168608    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:05:44.331467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:44.328574    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.360971 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:44.360929    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.485036 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:44.484927    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.485328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:44.381534    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.924537 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:44.718517    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 30 03:05:44.924537 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:44.860564    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.075582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.075527    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5" podUID="cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.117:8000/ready\": context deadline exceeded"
Oct 30 03:05:45.167628 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.167574    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.245507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.199574    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 03:05:45.245507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.200472    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 03:05:45.245507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.236414    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 03:05:45.245507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.236668    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 30 03:05:45.245507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.236886    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 03:05:45.245507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.239754    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" podUID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": context deadline exceeded"
Oct 30 03:05:45.355483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.354513    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 03:05:45.357451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.356397    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 03:05:45.359166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.359140    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:45.422997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.422960    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af4ef31-b040-46f4-82a5-709fb191bf31-kube-api-access-kbt88\" (UniqueName: \"kubernetes.io/projected/4af4ef31-b040-46f4-82a5-709fb191bf31-kube-api-access-kbt88\") pod \"4af4ef31-b040-46f4-82a5-709fb191bf31\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") "
Oct 30 03:05:45.423287 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.423261    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-xks-ca\") pod \"4af4ef31-b040-46f4-82a5-709fb191bf31\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") "
Oct 30 03:05:45.423593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.423547    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-openshift-ca\") pod \"4af4ef31-b040-46f4-82a5-709fb191bf31\" (UID: \"4af4ef31-b040-46f4-82a5-709fb191bf31\") "
Oct 30 03:05:45.424963 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.424937    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-openshift-ca" pod "4af4ef31-b040-46f4-82a5-709fb191bf31" (UID: "4af4ef31-b040-46f4-82a5-709fb191bf31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:45.520892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.485541    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.520892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.485954    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 03:05:45.520892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.486850    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.594697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.559215    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:05:45.641480 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.611571    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af4ef31-b040-46f4-82a5-709fb191bf31-kube-api-access-kbt88" pod "4af4ef31-b040-46f4-82a5-709fb191bf31" (UID: "4af4ef31-b040-46f4-82a5-709fb191bf31"). InnerVolumeSpecName "kube-api-access-kbt88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:45.708648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.708607    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-xks-ca" pod "4af4ef31-b040-46f4-82a5-709fb191bf31" (UID: "4af4ef31-b040-46f4-82a5-709fb191bf31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:05:45.725763 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.725719    4278 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.727139 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.726561    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.732458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.728555    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbt88\" (UniqueName: \"kubernetes.io/projected/4af4ef31-b040-46f4-82a5-709fb191bf31-kube-api-access-kbt88\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:05:45.732458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.728584    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af4ef31-b040-46f4-82a5-709fb191bf31-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:05:45.876568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:45.862327    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:46.057765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.057188    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 03:05:46.062475 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.058645    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 03:05:46.183509 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.157600    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 03:05:46.217468 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.190498    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5" podUID="cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.117:8000/ready\": context deadline exceeded"
Oct 30 03:05:46.217468 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.192525    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:05:46.217468 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.205744    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:46.262452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.251794    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:46.330679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.328539    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" podUID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:46.331281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.331226    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:46.442605 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.442520    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 03:05:46.446886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.446499    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 03:05:46.446886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.446551    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:05:46.446886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.446581    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 03:05:46.446886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.446611    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:05:46.446886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.446637    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 03:05:46.446886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.446665    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 03:05:46.446886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.446694    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 03:05:46.449591 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.449545    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn" event={"ID":"4af4ef31-b040-46f4-82a5-709fb191bf31","Type":"ContainerDied","Data":"b113e00063b002524adf5fee2c2ed401ad97c605895e101bf99ab826eca5aa94"}
Oct 30 03:05:46.449857 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.449840    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"
Oct 30 03:05:46.450347 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:46.450325    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b113e00063b002524adf5fee2c2ed401ad97c605895e101bf99ab826eca5aa94"
Oct 30 03:05:51.276891 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:51.276512    4278 patch_prober.go:29] interesting pod/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:05:51.276891 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:51.276594    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:53.107624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:53.107576    4278 patch_prober.go:29] interesting pod/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:05:53.108151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:53.107658    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:54.190829 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:05:54.190784    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:05:54.658489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.658443    4278 generic.go:356] "Generic (PLEG): container finished" podID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerID="97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a" exitCode=0
Oct 30 03:05:54.658677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.658522    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" event={"ID":"0e5c2e8d-d618-450e-81ff-04c418279c52","Type":"ContainerDied","Data":"97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a"}
Oct 30 03:05:54.659059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.659039    4278 scope.go:122] "RemoveContainer" containerID="97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a"
Oct 30 03:05:54.660436 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.660404    4278 generic.go:356] "Generic (PLEG): container finished" podID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerID="296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af" exitCode=0
Oct 30 03:05:54.660526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.660463    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" event={"ID":"340bdf62-58b6-4dc0-9814-d3a91b88e949","Type":"ContainerDied","Data":"296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af"}
Oct 30 03:05:54.660845 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.660831    4278 scope.go:122] "RemoveContainer" containerID="296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af"
Oct 30 03:05:54.661202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.661170    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:54.661273 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.661258    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:54.661325 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.661303    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:54.662395 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.662370    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:54.662483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.662450    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:54.662522 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:54.662484    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:55.704681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:55.703934    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" event={"ID":"340bdf62-58b6-4dc0-9814-d3a91b88e949","Type":"ContainerStarted","Data":"97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889"}
Oct 30 03:05:55.707178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:55.707135    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" event={"ID":"0e5c2e8d-d618-450e-81ff-04c418279c52","Type":"ContainerStarted","Data":"18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7"}
Oct 30 03:05:55.733446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:55.732742    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:05:55.733446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:55.732854    4278 patch_prober.go:29] interesting pod/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:05:55.733446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:55.732886    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 03:05:55.847288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:55.847259    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:05:55.847648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:55.847623    4278 patch_prober.go:29] interesting pod/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 03:05:55.847829 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:55.847803    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 03:05:56.729915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:56.729883    4278 patch_prober.go:29] interesting pod/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:05:56.730303 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:56.729932    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 03:05:56.847259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:56.847211    4278 patch_prober.go:29] interesting pod/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 03:05:56.847463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:56.847285    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 03:05:57.743618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:57.743572    4278 patch_prober.go:29] interesting pod/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:05:57.744112 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:57.743651    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 03:05:57.847608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:57.847569    4278 patch_prober.go:29] interesting pod/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 03:05:57.847793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:57.847634    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 03:05:58.744573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:58.742832    4278 patch_prober.go:29] interesting pod/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:05:58.744573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:58.742898    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 03:05:58.846965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:58.846932    4278 patch_prober.go:29] interesting pod/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 03:05:58.847138 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:58.846987    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 03:05:59.729557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:59.729519    4278 patch_prober.go:29] interesting pod/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:05:59.729742 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:59.729583    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 03:05:59.846749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:59.846715    4278 patch_prober.go:29] interesting pod/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 03:05:59.847129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:05:59.846767    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 03:06:00.742347 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:00.742306    4278 patch_prober.go:29] interesting pod/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:06:00.742697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:00.742653    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 03:06:00.849491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:00.849404    4278 patch_prober.go:29] interesting pod/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 03:06:00.856541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:00.850041    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 03:06:01.852648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:01.852614    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:06:01.853140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:01.853055    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:06:01.853925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:01.853904    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:06:01.864028 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:01.864002    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:06:01.864605 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:01.864582    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:06:01.865521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:01.865503    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:06:02.812832 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:02.812793    4278 scope.go:122] "RemoveContainer" containerID="ccca67314910d5b8b16a5fc89a85aa0fdbbb6c41593f86478cfd2c8596ce9d29"
Oct 30 03:06:03.478163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.478120    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"]
Oct 30 03:06:03.480753 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.480721    4278 cpu_manager.go:414] "containerMap: removing container" podUID="4af4ef31-b040-46f4-82a5-709fb191bf31" containerName="revoke-keys"
Oct 30 03:06:03.480753 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.480748    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af4ef31-b040-46f4-82a5-709fb191bf31" containerName="revoke-keys"
Oct 30 03:06:03.481267 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.481231    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af4ef31-b040-46f4-82a5-709fb191bf31" containerName="revoke-keys"
Oct 30 03:06:03.487454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.487413    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:03.505120 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.505092    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"]
Oct 30 03:06:03.506227 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.506206    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:06:03.506351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.506284    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:06:03.586324 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.586288    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:03.586509 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.586444    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvc4s\" (UniqueName: \"kubernetes.io/projected/0dde20c3-5c50-4865-9f5b-32f287664437-kube-api-access-mvc4s\") pod \"maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:03.586509 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.586494    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:03.687903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.687868    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:03.688088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.687973    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvc4s\" (UniqueName: \"kubernetes.io/projected/0dde20c3-5c50-4865-9f5b-32f287664437-kube-api-access-mvc4s\") pod \"maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:03.688088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.688010    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:03.688548 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.688528    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:03.690641 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.690606    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:03.707439 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.707402    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvc4s\" (UniqueName: \"kubernetes.io/projected/0dde20c3-5c50-4865-9f5b-32f287664437-kube-api-access-mvc4s\") pod \"maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:03.806316 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:03.806276    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:04.107581 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:04.107558    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"]
Oct 30 03:06:04.109788 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:06:04.109757    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dde20c3_5c50_4865_9f5b_32f287664437.slice/crio-9df01d6518a9cda08842e65adb0f81bb4d0efaab89f00fa2a32963278481aa5e WatchSource:0}: Error finding container 9df01d6518a9cda08842e65adb0f81bb4d0efaab89f00fa2a32963278481aa5e: Status 404 returned error can't find the container with id 9df01d6518a9cda08842e65adb0f81bb4d0efaab89f00fa2a32963278481aa5e
Oct 30 03:06:04.892701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:04.892668    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9" event={"ID":"0dde20c3-5c50-4865-9f5b-32f287664437","Type":"ContainerStarted","Data":"52a498cb726ee1776cb4bebb8bcf7c67b7a392838111fb6bc96afa9854327cb2"}
Oct 30 03:06:04.893109 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:04.892722    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9" event={"ID":"0dde20c3-5c50-4865-9f5b-32f287664437","Type":"ContainerStarted","Data":"9df01d6518a9cda08842e65adb0f81bb4d0efaab89f00fa2a32963278481aa5e"}
Oct 30 03:06:04.928461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:04.928384    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9" podStartSLOduration=1.928366834 podStartE2EDuration="1.928366834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:04.92615096 +0000 UTC m=+1502.850359143" watchObservedRunningTime="2026-10-30 03:06:04.928366834 +0000 UTC m=+1502.852575029"
Oct 30 03:06:05.901600 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:05.901567    4278 generic.go:356] "Generic (PLEG): container finished" podID="0dde20c3-5c50-4865-9f5b-32f287664437" containerID="52a498cb726ee1776cb4bebb8bcf7c67b7a392838111fb6bc96afa9854327cb2" exitCode=0
Oct 30 03:06:05.902001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:05.901658    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9" event={"ID":"0dde20c3-5c50-4865-9f5b-32f287664437","Type":"ContainerDied","Data":"52a498cb726ee1776cb4bebb8bcf7c67b7a392838111fb6bc96afa9854327cb2"}
Oct 30 03:06:07.094796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.094770    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:07.142957 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.142923    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-openshift-ca\") pod \"0dde20c3-5c50-4865-9f5b-32f287664437\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") "
Oct 30 03:06:07.143133 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.142980    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dde20c3-5c50-4865-9f5b-32f287664437-kube-api-access-mvc4s\" (UniqueName: \"kubernetes.io/projected/0dde20c3-5c50-4865-9f5b-32f287664437-kube-api-access-mvc4s\") pod \"0dde20c3-5c50-4865-9f5b-32f287664437\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") "
Oct 30 03:06:07.143133 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.143052    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-xks-ca\") pod \"0dde20c3-5c50-4865-9f5b-32f287664437\" (UID: \"0dde20c3-5c50-4865-9f5b-32f287664437\") "
Oct 30 03:06:07.143367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.143339    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-openshift-ca" pod "0dde20c3-5c50-4865-9f5b-32f287664437" (UID: "0dde20c3-5c50-4865-9f5b-32f287664437"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:07.144322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.144292    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:06:07.145393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.145361    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dde20c3-5c50-4865-9f5b-32f287664437-kube-api-access-mvc4s" pod "0dde20c3-5c50-4865-9f5b-32f287664437" (UID: "0dde20c3-5c50-4865-9f5b-32f287664437"). InnerVolumeSpecName "kube-api-access-mvc4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:07.145508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.145474    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-xks-ca" pod "0dde20c3-5c50-4865-9f5b-32f287664437" (UID: "0dde20c3-5c50-4865-9f5b-32f287664437"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:06:07.246582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.246499    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvc4s\" (UniqueName: \"kubernetes.io/projected/0dde20c3-5c50-4865-9f5b-32f287664437-kube-api-access-mvc4s\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:06:07.246582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.246535    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dde20c3-5c50-4865-9f5b-32f287664437-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:06:07.919837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.919807    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"
Oct 30 03:06:07.919837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.919833    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9" event={"ID":"0dde20c3-5c50-4865-9f5b-32f287664437","Type":"ContainerDied","Data":"9df01d6518a9cda08842e65adb0f81bb4d0efaab89f00fa2a32963278481aa5e"}
Oct 30 03:06:07.920066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:07.919881    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9df01d6518a9cda08842e65adb0f81bb4d0efaab89f00fa2a32963278481aa5e"
Oct 30 03:06:24.190616 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:06:24.190577    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:06:25.206662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:25.206607    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:06:27.992007 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:27.991889    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"]
Oct 30 03:06:27.997790 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:27.997761    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-cxrxj"]
Oct 30 03:06:28.237416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:28.237369    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49bddf5-3955-4a4d-ad6a-a432033350ee" path="/var/lib/kubelet/pods/e49bddf5-3955-4a4d-ad6a-a432033350ee/volumes"
Oct 30 03:06:41.240986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.240943    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"]
Oct 30 03:06:41.241655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.241640    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0dde20c3-5c50-4865-9f5b-32f287664437" containerName="revoke-keys"
Oct 30 03:06:41.241699 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.241656    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dde20c3-5c50-4865-9f5b-32f287664437" containerName="revoke-keys"
Oct 30 03:06:41.241803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.241794    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dde20c3-5c50-4865-9f5b-32f287664437" containerName="revoke-keys"
Oct 30 03:06:41.246892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.246868    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:41.290316 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.290277    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"]
Oct 30 03:06:41.290567 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.290547    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:06:41.290649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.290618    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:06:41.420073 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.420020    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4xs\" (UniqueName: \"kubernetes.io/projected/060e5d53-3959-4a80-b3e2-7e8e171b174b-kube-api-access-7p4xs\") pod \"maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:41.420285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.420079    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:41.420285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.420133    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:41.537452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.533999    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4xs\" (UniqueName: \"kubernetes.io/projected/060e5d53-3959-4a80-b3e2-7e8e171b174b-kube-api-access-7p4xs\") pod \"maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:41.602705 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.602665    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:41.617609 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.603688    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:41.617974 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.617949    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:41.630545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.630236    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:41.662768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.650204    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4xs\" (UniqueName: \"kubernetes.io/projected/060e5d53-3959-4a80-b3e2-7e8e171b174b-kube-api-access-7p4xs\") pod \"maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:41.872153 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:41.872060    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:42.209779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:42.209752    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"]
Oct 30 03:06:42.243254 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:42.243221    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr" event={"ID":"060e5d53-3959-4a80-b3e2-7e8e171b174b","Type":"ContainerStarted","Data":"3ca3d060777f26cd20384937560ac2513c4f0783d5191121ae42d4c01a948a5e"}
Oct 30 03:06:43.253211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:43.253162    4278 generic.go:356] "Generic (PLEG): container finished" podID="060e5d53-3959-4a80-b3e2-7e8e171b174b" containerID="4545cd9e4f317415951a14725995ca885a9f76569251b374b572b2f1e6fcd8a0" exitCode=0
Oct 30 03:06:43.253727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:43.253242    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr" event={"ID":"060e5d53-3959-4a80-b3e2-7e8e171b174b","Type":"ContainerDied","Data":"4545cd9e4f317415951a14725995ca885a9f76569251b374b572b2f1e6fcd8a0"}
Oct 30 03:06:44.449942 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:44.449916    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:44.480296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:44.480272    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-xks-ca\") pod \"060e5d53-3959-4a80-b3e2-7e8e171b174b\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") "
Oct 30 03:06:44.480465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:44.480361    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-openshift-ca\") pod \"060e5d53-3959-4a80-b3e2-7e8e171b174b\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") "
Oct 30 03:06:44.480548 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:44.480485    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060e5d53-3959-4a80-b3e2-7e8e171b174b-kube-api-access-7p4xs\" (UniqueName: \"kubernetes.io/projected/060e5d53-3959-4a80-b3e2-7e8e171b174b-kube-api-access-7p4xs\") pod \"060e5d53-3959-4a80-b3e2-7e8e171b174b\" (UID: \"060e5d53-3959-4a80-b3e2-7e8e171b174b\") "
Oct 30 03:06:44.480741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:44.480713    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-openshift-ca" pod "060e5d53-3959-4a80-b3e2-7e8e171b174b" (UID: "060e5d53-3959-4a80-b3e2-7e8e171b174b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:44.481101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:44.481085    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:06:44.482440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:44.482408    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-xks-ca" pod "060e5d53-3959-4a80-b3e2-7e8e171b174b" (UID: "060e5d53-3959-4a80-b3e2-7e8e171b174b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:06:44.482544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:44.482524    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060e5d53-3959-4a80-b3e2-7e8e171b174b-kube-api-access-7p4xs" pod "060e5d53-3959-4a80-b3e2-7e8e171b174b" (UID: "060e5d53-3959-4a80-b3e2-7e8e171b174b"). InnerVolumeSpecName "kube-api-access-7p4xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:44.582523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:44.582415    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p4xs\" (UniqueName: \"kubernetes.io/projected/060e5d53-3959-4a80-b3e2-7e8e171b174b-kube-api-access-7p4xs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:06:44.582523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:44.582464    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060e5d53-3959-4a80-b3e2-7e8e171b174b-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:06:45.273759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:45.273723    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"
Oct 30 03:06:45.273759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:45.273756    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr" event={"ID":"060e5d53-3959-4a80-b3e2-7e8e171b174b","Type":"ContainerDied","Data":"3ca3d060777f26cd20384937560ac2513c4f0783d5191121ae42d4c01a948a5e"}
Oct 30 03:06:45.274121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:06:45.273806    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ca3d060777f26cd20384937560ac2513c4f0783d5191121ae42d4c01a948a5e"
Oct 30 03:06:54.192630 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:06:54.192589    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:07:02.899386 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:02.899351    4278 scope.go:122] "RemoveContainer" containerID="95e7079d381ae5e41d5a28efe6d01c9ac24843ae66b45688da071c6d2ecfbc8d"
Oct 30 03:07:02.910111 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:02.910088    4278 scope.go:122] "RemoveContainer" containerID="ab456413fd34d7422604d5aef3a0b5e4bbadf60bd590e763729039bdef04f75e"
Oct 30 03:07:02.920254 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:02.920230    4278 scope.go:122] "RemoveContainer" containerID="4babcf2b5ba5fb84e60a257d46738b72b33ac70239464e0ae29a9a4b54f72540"
Oct 30 03:07:02.930573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:02.930550    4278 scope.go:122] "RemoveContainer" containerID="3a6b516a5457a4acf19f49b15e53a309fdecac4d8a3bbc3ef6744b8f7ddf6d69"
Oct 30 03:07:02.941032 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:02.941008    4278 scope.go:122] "RemoveContainer" containerID="c32ed659c0ed56395cb46091f45e88b96ee200927555c74447fb16339a1bc8b4"
Oct 30 03:07:02.950876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:02.950855    4278 scope.go:122] "RemoveContainer" containerID="bfdfecbcad6ce5322220e292c4d983715bcc38290f13d1815e848eb9d4a785bc"
Oct 30 03:07:02.963766 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:02.963741    4278 scope.go:122] "RemoveContainer" containerID="8384c36aaf6a221dbaaac4a789d7bd38113846e0af5ed7cbb9291b954f33b3ac"
Oct 30 03:07:04.827093 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:04.827056    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"]
Oct 30 03:07:04.827933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:04.827912    4278 cpu_manager.go:414] "containerMap: removing container" podUID="060e5d53-3959-4a80-b3e2-7e8e171b174b" containerName="revoke-keys"
Oct 30 03:07:04.827980 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:04.827935    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060e5d53-3959-4a80-b3e2-7e8e171b174b" containerName="revoke-keys"
Oct 30 03:07:04.828057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:04.828048    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="060e5d53-3959-4a80-b3e2-7e8e171b174b" containerName="revoke-keys"
Oct 30 03:07:04.833928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:04.833907    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:04.850566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:04.850536    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"]
Oct 30 03:07:04.856709 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:04.856676    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:07:04.856848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:04.856775    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:07:05.011496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.011458    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866f6ca6-f8j2p\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:05.011658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.011514    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvpgn\" (UniqueName: \"kubernetes.io/projected/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-kube-api-access-fvpgn\") pod \"maas-api-revoke-sub-866f6ca6-f8j2p\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:05.011658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.011624    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866f6ca6-f8j2p\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:05.113720 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.113632    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866f6ca6-f8j2p\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:05.113720 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.113683    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvpgn\" (UniqueName: \"kubernetes.io/projected/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-kube-api-access-fvpgn\") pod \"maas-api-revoke-sub-866f6ca6-f8j2p\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:05.113720 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.113716    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866f6ca6-f8j2p\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:05.114385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.114358    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866f6ca6-f8j2p\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:05.116134 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.116112    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866f6ca6-f8j2p\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:05.132341 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.132322    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvpgn\" (UniqueName: \"kubernetes.io/projected/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-kube-api-access-fvpgn\") pod \"maas-api-revoke-sub-866f6ca6-f8j2p\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:05.148674 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.148652    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:05.653693 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:05.653664    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"]
Oct 30 03:07:05.655050 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:07:05.655010    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3faf8cf9_2bb5_4863_9740_b907ced8f2cd.slice/crio-c71e7aad29bc0ff249da32bcd87fc546ac4de77a9461300e95a2615260cd9367 WatchSource:0}: Error finding container c71e7aad29bc0ff249da32bcd87fc546ac4de77a9461300e95a2615260cd9367: Status 404 returned error can't find the container with id c71e7aad29bc0ff249da32bcd87fc546ac4de77a9461300e95a2615260cd9367
Oct 30 03:07:06.650149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:06.650110    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p" event={"ID":"3faf8cf9-2bb5-4863-9740-b907ced8f2cd","Type":"ContainerStarted","Data":"67c1559199191ed9f37a3b8d41faa9f358e7e071bc9c6c6f3aa89a418dd72568"}
Oct 30 03:07:06.650612 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:06.650197    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p" event={"ID":"3faf8cf9-2bb5-4863-9740-b907ced8f2cd","Type":"ContainerStarted","Data":"c71e7aad29bc0ff249da32bcd87fc546ac4de77a9461300e95a2615260cd9367"}
Oct 30 03:07:06.689136 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:06.689083    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p" podStartSLOduration=2.689070089 podStartE2EDuration="2.689070089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:06.688124418 +0000 UTC m=+1564.612332607" watchObservedRunningTime="2026-10-30 03:07:06.689070089 +0000 UTC m=+1564.613278280"
Oct 30 03:07:07.659612 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:07.659580    4278 generic.go:356] "Generic (PLEG): container finished" podID="3faf8cf9-2bb5-4863-9740-b907ced8f2cd" containerID="67c1559199191ed9f37a3b8d41faa9f358e7e071bc9c6c6f3aa89a418dd72568" exitCode=0
Oct 30 03:07:07.660047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:07.659686    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p" event={"ID":"3faf8cf9-2bb5-4863-9740-b907ced8f2cd","Type":"ContainerDied","Data":"67c1559199191ed9f37a3b8d41faa9f358e7e071bc9c6c6f3aa89a418dd72568"}
Oct 30 03:07:08.838486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:08.838463    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:08.921449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:08.921302    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-xks-ca\") pod \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") "
Oct 30 03:07:08.921449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:08.921377    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-kube-api-access-fvpgn\" (UniqueName: \"kubernetes.io/projected/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-kube-api-access-fvpgn\") pod \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") "
Oct 30 03:07:08.921449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:08.921437    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-openshift-ca\") pod \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\" (UID: \"3faf8cf9-2bb5-4863-9740-b907ced8f2cd\") "
Oct 30 03:07:08.922405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:08.922075    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-openshift-ca" pod "3faf8cf9-2bb5-4863-9740-b907ced8f2cd" (UID: "3faf8cf9-2bb5-4863-9740-b907ced8f2cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:08.922644 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:08.922579    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:08.924104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:08.924030    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-kube-api-access-fvpgn" pod "3faf8cf9-2bb5-4863-9740-b907ced8f2cd" (UID: "3faf8cf9-2bb5-4863-9740-b907ced8f2cd"). InnerVolumeSpecName "kube-api-access-fvpgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:08.924382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:08.924357    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-xks-ca" pod "3faf8cf9-2bb5-4863-9740-b907ced8f2cd" (UID: "3faf8cf9-2bb5-4863-9740-b907ced8f2cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:07:09.024155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.024121    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:09.024155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.024148    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvpgn\" (UniqueName: \"kubernetes.io/projected/3faf8cf9-2bb5-4863-9740-b907ced8f2cd-kube-api-access-fvpgn\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:09.572414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.572375    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"]
Oct 30 03:07:09.573170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.573155    4278 cpu_manager.go:414] "containerMap: removing container" podUID="3faf8cf9-2bb5-4863-9740-b907ced8f2cd" containerName="revoke-keys"
Oct 30 03:07:09.573213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.573172    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3faf8cf9-2bb5-4863-9740-b907ced8f2cd" containerName="revoke-keys"
Oct 30 03:07:09.573287 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.573278    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="3faf8cf9-2bb5-4863-9740-b907ced8f2cd" containerName="revoke-keys"
Oct 30 03:07:09.578136 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.578092    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:09.599549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.599514    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"]
Oct 30 03:07:09.635116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.635070    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gbq9\" (UniqueName: \"kubernetes.io/projected/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-kube-api-access-2gbq9\") pod \"maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:09.635617 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.635580    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:09.635768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.635754    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:09.685247 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.685217    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"
Oct 30 03:07:09.685247 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.685231    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p" event={"ID":"3faf8cf9-2bb5-4863-9740-b907ced8f2cd","Type":"ContainerDied","Data":"c71e7aad29bc0ff249da32bcd87fc546ac4de77a9461300e95a2615260cd9367"}
Oct 30 03:07:09.685543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.685277    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c71e7aad29bc0ff249da32bcd87fc546ac4de77a9461300e95a2615260cd9367"
Oct 30 03:07:09.737160 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.737127    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:09.737383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.737363    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gbq9\" (UniqueName: \"kubernetes.io/projected/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-kube-api-access-2gbq9\") pod \"maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:09.737538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.737523    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:09.737797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.737776    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:09.739921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.739897    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:09.757234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.757206    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gbq9\" (UniqueName: \"kubernetes.io/projected/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-kube-api-access-2gbq9\") pod \"maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:09.902262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:09.902160    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:10.208248 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:10.208218    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"]
Oct 30 03:07:10.210756 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:07:10.210725    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c6f3d6_e1cd_47a5_b2d6_af21e904408a.slice/crio-cbf4e63f273e268a50e572215547877c321236ab4a5f310cf5aad1c9f9cbf244 WatchSource:0}: Error finding container cbf4e63f273e268a50e572215547877c321236ab4a5f310cf5aad1c9f9cbf244: Status 404 returned error can't find the container with id cbf4e63f273e268a50e572215547877c321236ab4a5f310cf5aad1c9f9cbf244
Oct 30 03:07:10.694611 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:10.694573    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29" event={"ID":"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a","Type":"ContainerStarted","Data":"a57a818e1b6fd585271b4bf1865235a127aeeb8ad82dfdc1c25b94b56a8e2475"}
Oct 30 03:07:10.694754 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:10.694672    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29" event={"ID":"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a","Type":"ContainerStarted","Data":"cbf4e63f273e268a50e572215547877c321236ab4a5f310cf5aad1c9f9cbf244"}
Oct 30 03:07:10.734593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:10.734370    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29" podStartSLOduration=1.734352997 podStartE2EDuration="1.734352997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:10.733589875 +0000 UTC m=+1568.657798064" watchObservedRunningTime="2026-10-30 03:07:10.734352997 +0000 UTC m=+1568.658561189"
Oct 30 03:07:11.704214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:11.704128    4278 generic.go:356] "Generic (PLEG): container finished" podID="e0c6f3d6-e1cd-47a5-b2d6-af21e904408a" containerID="a57a818e1b6fd585271b4bf1865235a127aeeb8ad82dfdc1c25b94b56a8e2475" exitCode=0
Oct 30 03:07:11.704214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:11.704157    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29" event={"ID":"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a","Type":"ContainerDied","Data":"a57a818e1b6fd585271b4bf1865235a127aeeb8ad82dfdc1c25b94b56a8e2475"}
Oct 30 03:07:12.880890 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:12.880864    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:12.992251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:12.992165    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-xks-ca\") pod \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") "
Oct 30 03:07:12.992469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:12.992282    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-openshift-ca\") pod \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") "
Oct 30 03:07:12.992469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:12.992383    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-kube-api-access-2gbq9\" (UniqueName: \"kubernetes.io/projected/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-kube-api-access-2gbq9\") pod \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\" (UID: \"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a\") "
Oct 30 03:07:12.992729 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:12.992702    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-openshift-ca" pod "e0c6f3d6-e1cd-47a5-b2d6-af21e904408a" (UID: "e0c6f3d6-e1cd-47a5-b2d6-af21e904408a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:12.993271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:12.993248    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:12.994517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:12.994493    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-xks-ca" pod "e0c6f3d6-e1cd-47a5-b2d6-af21e904408a" (UID: "e0c6f3d6-e1cd-47a5-b2d6-af21e904408a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:07:12.994517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:12.994502    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-kube-api-access-2gbq9" pod "e0c6f3d6-e1cd-47a5-b2d6-af21e904408a" (UID: "e0c6f3d6-e1cd-47a5-b2d6-af21e904408a"). InnerVolumeSpecName "kube-api-access-2gbq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:13.095202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:13.095174    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:13.095202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:13.095200    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gbq9\" (UniqueName: \"kubernetes.io/projected/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a-kube-api-access-2gbq9\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:13.744129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:13.744076    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29" event={"ID":"e0c6f3d6-e1cd-47a5-b2d6-af21e904408a","Type":"ContainerDied","Data":"cbf4e63f273e268a50e572215547877c321236ab4a5f310cf5aad1c9f9cbf244"}
Oct 30 03:07:13.744363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:13.744170    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf4e63f273e268a50e572215547877c321236ab4a5f310cf5aad1c9f9cbf244"
Oct 30 03:07:13.744363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:13.744255    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"
Oct 30 03:07:17.678596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:17.678504    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"]
Oct 30 03:07:17.679086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:17.678729    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs" podUID="0acc2e64-eb9a-411f-8cb4-e169b4edf58d" containerName="maas-api" containerID="cri-o://074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340" gracePeriod=30
Oct 30 03:07:18.004242 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.004210    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:07:18.155907 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.151310    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-maas-api-tls\") pod \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") "
Oct 30 03:07:18.161959 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.156364    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-kube-api-access-lz85r\" (UniqueName: \"kubernetes.io/projected/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-kube-api-access-lz85r\") pod \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") "
Oct 30 03:07:18.162704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.162664    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-metrics-certs\") pod \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") "
Oct 30 03:07:18.178150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.175785    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-openshift-service-ca\") pod \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\" (UID: \"0acc2e64-eb9a-411f-8cb4-e169b4edf58d\") "
Oct 30 03:07:18.314455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.301260    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-openshift-service-ca" pod "0acc2e64-eb9a-411f-8cb4-e169b4edf58d" (UID: "0acc2e64-eb9a-411f-8cb4-e169b4edf58d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:18.332847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.332811    4278 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-openshift-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:18.362030 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.361689    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-maas-api-tls" pod "0acc2e64-eb9a-411f-8cb4-e169b4edf58d" (UID: "0acc2e64-eb9a-411f-8cb4-e169b4edf58d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:07:18.415362 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.415312    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-kube-api-access-lz85r" pod "0acc2e64-eb9a-411f-8cb4-e169b4edf58d" (UID: "0acc2e64-eb9a-411f-8cb4-e169b4edf58d"). InnerVolumeSpecName "kube-api-access-lz85r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:18.436448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.431838    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-metrics-certs" pod "0acc2e64-eb9a-411f-8cb4-e169b4edf58d" (UID: "0acc2e64-eb9a-411f-8cb4-e169b4edf58d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:07:18.438325 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.438286    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-maas-api-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:18.438628 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.438609    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz85r\" (UniqueName: \"kubernetes.io/projected/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-kube-api-access-lz85r\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:18.438731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.438718    4278 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0acc2e64-eb9a-411f-8cb4-e169b4edf58d-metrics-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:18.669621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.669420    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"]
Oct 30 03:07:18.686204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.679723    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ffb1886-msnvr"]
Oct 30 03:07:18.796583 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.793951    4278 generic.go:356] "Generic (PLEG): container finished" podID="0acc2e64-eb9a-411f-8cb4-e169b4edf58d" containerID="074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340" exitCode=0
Oct 30 03:07:18.796583 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.794082    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs" event={"ID":"0acc2e64-eb9a-411f-8cb4-e169b4edf58d","Type":"ContainerDied","Data":"074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340"}
Oct 30 03:07:18.796583 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.794142    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs" event={"ID":"0acc2e64-eb9a-411f-8cb4-e169b4edf58d","Type":"ContainerDied","Data":"dae13785b41299812d938e554f923d1c4a5d74658912a2955f3925b56c5206a5"}
Oct 30 03:07:18.796583 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.794174    4278 scope.go:122] "RemoveContainer" containerID="074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340"
Oct 30 03:07:18.796583 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.794385    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"
Oct 30 03:07:18.813036 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.813006    4278 scope.go:122] "RemoveContainer" containerID="074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340"
Oct 30 03:07:18.813505 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:07:18.813470    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340\": container with ID starting with 074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340 not found: ID does not exist" containerID="074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340"
Oct 30 03:07:18.813645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.813508    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340"} err="failed to get container status \"074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340\": rpc error: code = NotFound desc = could not find container \"074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340\": container with ID starting with 074fcf82ed98b3f3c2f01a792ac08647f46b94e6b3deb04bdc5ea35f55e0c340 not found: ID does not exist"
Oct 30 03:07:18.856128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.856087    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"]
Oct 30 03:07:18.866801 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:18.866766    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ffb1886-869d5bb9dc-45bgs"]
Oct 30 03:07:20.201963 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:20.201927    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060e5d53-3959-4a80-b3e2-7e8e171b174b" path="/var/lib/kubelet/pods/060e5d53-3959-4a80-b3e2-7e8e171b174b/volumes"
Oct 30 03:07:20.202498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:20.202482    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0acc2e64-eb9a-411f-8cb4-e169b4edf58d" path="/var/lib/kubelet/pods/0acc2e64-eb9a-411f-8cb4-e169b4edf58d/volumes"
Oct 30 03:07:20.549693 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:20.549655    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"]
Oct 30 03:07:20.550191 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:20.550145    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh" podUID="a14e4038-6cba-41c3-9736-9a9a0e8194e5" containerName="maas-api" containerID="cri-o://3e239b5fe4f12f9a8b092a486ac0a7cb38e04de4448fb559a29e5549f0cd5b08" gracePeriod=30
Oct 30 03:07:20.873563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:20.873459    4278 generic.go:356] "Generic (PLEG): container finished" podID="a14e4038-6cba-41c3-9736-9a9a0e8194e5" containerID="3e239b5fe4f12f9a8b092a486ac0a7cb38e04de4448fb559a29e5549f0cd5b08" exitCode=0
Oct 30 03:07:20.873910 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:20.873875    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh" event={"ID":"a14e4038-6cba-41c3-9736-9a9a0e8194e5","Type":"ContainerDied","Data":"3e239b5fe4f12f9a8b092a486ac0a7cb38e04de4448fb559a29e5549f0cd5b08"}
Oct 30 03:07:21.520905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.520861    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:07:21.592447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.591890    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14e4038-6cba-41c3-9736-9a9a0e8194e5-kube-api-access-shxpm\" (UniqueName: \"kubernetes.io/projected/a14e4038-6cba-41c3-9736-9a9a0e8194e5-kube-api-access-shxpm\") pod \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") "
Oct 30 03:07:21.592447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.591946    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-maas-api-tls\") pod \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") "
Oct 30 03:07:21.592447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.591985    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-metrics-certs\") pod \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") "
Oct 30 03:07:21.592764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.592450    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a14e4038-6cba-41c3-9736-9a9a0e8194e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a14e4038-6cba-41c3-9736-9a9a0e8194e5-openshift-service-ca\") pod \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\" (UID: \"a14e4038-6cba-41c3-9736-9a9a0e8194e5\") "
Oct 30 03:07:21.598890 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.598765    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14e4038-6cba-41c3-9736-9a9a0e8194e5-openshift-service-ca" pod "a14e4038-6cba-41c3-9736-9a9a0e8194e5" (UID: "a14e4038-6cba-41c3-9736-9a9a0e8194e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:21.608923 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.608883    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-metrics-certs" pod "a14e4038-6cba-41c3-9736-9a9a0e8194e5" (UID: "a14e4038-6cba-41c3-9736-9a9a0e8194e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:07:21.609283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.609234    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14e4038-6cba-41c3-9736-9a9a0e8194e5-kube-api-access-shxpm" pod "a14e4038-6cba-41c3-9736-9a9a0e8194e5" (UID: "a14e4038-6cba-41c3-9736-9a9a0e8194e5"). InnerVolumeSpecName "kube-api-access-shxpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:21.612059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.612014    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-maas-api-tls" pod "a14e4038-6cba-41c3-9736-9a9a0e8194e5" (UID: "a14e4038-6cba-41c3-9736-9a9a0e8194e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:07:21.616494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.616408    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"]
Oct 30 03:07:21.624508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.623688    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d7fde0-hspv9"]
Oct 30 03:07:21.697903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.697805    4278 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a14e4038-6cba-41c3-9736-9a9a0e8194e5-openshift-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:21.697903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.697849    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shxpm\" (UniqueName: \"kubernetes.io/projected/a14e4038-6cba-41c3-9736-9a9a0e8194e5-kube-api-access-shxpm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:21.697903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.697867    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-maas-api-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:21.697903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.697884    4278 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a14e4038-6cba-41c3-9736-9a9a0e8194e5-metrics-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:21.882402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.882370    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"
Oct 30 03:07:21.882643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.882364    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh" event={"ID":"a14e4038-6cba-41c3-9736-9a9a0e8194e5","Type":"ContainerDied","Data":"9ac815f2be908263b6b0db70ae354484fae20da72a358e555ebb6fd68739a862"}
Oct 30 03:07:21.882643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.882520    4278 scope.go:122] "RemoveContainer" containerID="3e239b5fe4f12f9a8b092a486ac0a7cb38e04de4448fb559a29e5549f0cd5b08"
Oct 30 03:07:21.942209 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.942178    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"]
Oct 30 03:07:21.946277 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:21.946246    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh"]
Oct 30 03:07:22.203194 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:22.203158    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dde20c3-5c50-4865-9f5b-32f287664437" path="/var/lib/kubelet/pods/0dde20c3-5c50-4865-9f5b-32f287664437/volumes"
Oct 30 03:07:22.203720 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:22.203705    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a14e4038-6cba-41c3-9736-9a9a0e8194e5" path="/var/lib/kubelet/pods/a14e4038-6cba-41c3-9736-9a9a0e8194e5/volumes"
Oct 30 03:07:23.864824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:23.864777    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fde0-79d9498546-5hcnh" podUID="a14e4038-6cba-41c3-9736-9a9a0e8194e5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.116:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:24.191380 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:07:24.191291    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:07:32.373449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.368057    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"]
Oct 30 03:07:32.373449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.368286    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy" containerID="cri-o://6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4" gracePeriod=30
Oct 30 03:07:32.867585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.867543    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"]
Oct 30 03:07:32.868352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.868330    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a14e4038-6cba-41c3-9736-9a9a0e8194e5" containerName="maas-api"
Oct 30 03:07:32.868481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.868353    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14e4038-6cba-41c3-9736-9a9a0e8194e5" containerName="maas-api"
Oct 30 03:07:32.868481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.868365    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0acc2e64-eb9a-411f-8cb4-e169b4edf58d" containerName="maas-api"
Oct 30 03:07:32.868481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.868372    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0acc2e64-eb9a-411f-8cb4-e169b4edf58d" containerName="maas-api"
Oct 30 03:07:32.868481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.868401    4278 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6f3d6-e1cd-47a5-b2d6-af21e904408a" containerName="revoke-keys"
Oct 30 03:07:32.868481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.868409    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6f3d6-e1cd-47a5-b2d6-af21e904408a" containerName="revoke-keys"
Oct 30 03:07:32.868743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.868594    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6f3d6-e1cd-47a5-b2d6-af21e904408a" containerName="revoke-keys"
Oct 30 03:07:32.868743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.868623    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14e4038-6cba-41c3-9736-9a9a0e8194e5" containerName="maas-api"
Oct 30 03:07:32.868743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.868632    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0acc2e64-eb9a-411f-8cb4-e169b4edf58d" containerName="maas-api"
Oct 30 03:07:32.872611 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.872587    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:32.889434 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.889395    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:07:32.889588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.889526    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:07:32.898828 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.898804    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"]
Oct 30 03:07:32.917056 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.917004    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:32.917319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.917284    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:32.917399 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:32.917356    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxtqq\" (UniqueName: \"kubernetes.io/projected/a61bfe15-f42d-424a-9376-37ec7d48581d-kube-api-access-rxtqq\") pod \"maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:33.018688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:33.018652    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:33.018905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:33.018804    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:33.018905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:33.018855    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxtqq\" (UniqueName: \"kubernetes.io/projected/a61bfe15-f42d-424a-9376-37ec7d48581d-kube-api-access-rxtqq\") pod \"maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:33.019574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:33.019548    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:33.021048 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:33.021022    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:33.040980 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:33.040939    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxtqq\" (UniqueName: \"kubernetes.io/projected/a61bfe15-f42d-424a-9376-37ec7d48581d-kube-api-access-rxtqq\") pod \"maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:33.189274 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:33.187976    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:33.572492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:33.572455    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"]
Oct 30 03:07:33.575500 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:07:33.575470    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61bfe15_f42d_424a_9376_37ec7d48581d.slice/crio-168f01839e1e2e5354ee11b09e503d39a9ec86381c26c64679b705a84bc666bb WatchSource:0}: Error finding container 168f01839e1e2e5354ee11b09e503d39a9ec86381c26c64679b705a84bc666bb: Status 404 returned error can't find the container with id 168f01839e1e2e5354ee11b09e503d39a9ec86381c26c64679b705a84bc666bb
Oct 30 03:07:34.004332 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:34.004303    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk" event={"ID":"a61bfe15-f42d-424a-9376-37ec7d48581d","Type":"ContainerStarted","Data":"c2cc6eebc68b51ad66dd283e18e6ce117453c58fba45d8c04d7a7e64e5978c68"}
Oct 30 03:07:34.004516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:34.004357    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk" event={"ID":"a61bfe15-f42d-424a-9376-37ec7d48581d","Type":"ContainerStarted","Data":"168f01839e1e2e5354ee11b09e503d39a9ec86381c26c64679b705a84bc666bb"}
Oct 30 03:07:34.050724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:34.050674    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk" podStartSLOduration=2.05065991 podStartE2EDuration="2.05065991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:34.046213702 +0000 UTC m=+1591.970421891" watchObservedRunningTime="2026-10-30 03:07:34.05065991 +0000 UTC m=+1591.974868098"
Oct 30 03:07:35.013972 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:35.013937    4278 generic.go:356] "Generic (PLEG): container finished" podID="a61bfe15-f42d-424a-9376-37ec7d48581d" containerID="c2cc6eebc68b51ad66dd283e18e6ce117453c58fba45d8c04d7a7e64e5978c68" exitCode=0
Oct 30 03:07:35.014366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:35.014014    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk" event={"ID":"a61bfe15-f42d-424a-9376-37ec7d48581d","Type":"ContainerDied","Data":"c2cc6eebc68b51ad66dd283e18e6ce117453c58fba45d8c04d7a7e64e5978c68"}
Oct 30 03:07:36.212536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:36.212509    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:36.313824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:36.313733    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-openshift-ca\") pod \"a61bfe15-f42d-424a-9376-37ec7d48581d\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") "
Oct 30 03:07:36.313824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:36.313821    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61bfe15-f42d-424a-9376-37ec7d48581d-kube-api-access-rxtqq\" (UniqueName: \"kubernetes.io/projected/a61bfe15-f42d-424a-9376-37ec7d48581d-kube-api-access-rxtqq\") pod \"a61bfe15-f42d-424a-9376-37ec7d48581d\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") "
Oct 30 03:07:36.314009 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:36.313874    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-xks-ca\") pod \"a61bfe15-f42d-424a-9376-37ec7d48581d\" (UID: \"a61bfe15-f42d-424a-9376-37ec7d48581d\") "
Oct 30 03:07:36.314188 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:36.314164    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-openshift-ca" pod "a61bfe15-f42d-424a-9376-37ec7d48581d" (UID: "a61bfe15-f42d-424a-9376-37ec7d48581d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:36.314595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:36.314575    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:36.315997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:36.315970    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-xks-ca" pod "a61bfe15-f42d-424a-9376-37ec7d48581d" (UID: "a61bfe15-f42d-424a-9376-37ec7d48581d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:07:36.316109 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:36.316084    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61bfe15-f42d-424a-9376-37ec7d48581d-kube-api-access-rxtqq" pod "a61bfe15-f42d-424a-9376-37ec7d48581d" (UID: "a61bfe15-f42d-424a-9376-37ec7d48581d"). InnerVolumeSpecName "kube-api-access-rxtqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:36.416323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:36.416284    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxtqq\" (UniqueName: \"kubernetes.io/projected/a61bfe15-f42d-424a-9376-37ec7d48581d-kube-api-access-rxtqq\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:36.416323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:36.416323    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61bfe15-f42d-424a-9376-37ec7d48581d-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:37.031950 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.031919    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"
Oct 30 03:07:37.031950 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.031928    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk" event={"ID":"a61bfe15-f42d-424a-9376-37ec7d48581d","Type":"ContainerDied","Data":"168f01839e1e2e5354ee11b09e503d39a9ec86381c26c64679b705a84bc666bb"}
Oct 30 03:07:37.032174 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.031974    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="168f01839e1e2e5354ee11b09e503d39a9ec86381c26c64679b705a84bc666bb"
Oct 30 03:07:37.644124 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.644095    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:07:37.737487 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.737394    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-data\") pod \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") "
Oct 30 03:07:37.737487 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.737451    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-credential-socket\") pod \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") "
Oct 30 03:07:37.738458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.737524    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-podinfo\") pod \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") "
Oct 30 03:07:37.738458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.737558    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-token\" (UniqueName: \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-token\") pod \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") "
Oct 30 03:07:37.738458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.737613    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-certs\") pod \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") "
Oct 30 03:07:37.738458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.737686    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-data" pod "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" (UID: "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:37.738458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.737793    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-credential-socket" pod "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" (UID: "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:37.738458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.737900    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-certs" pod "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" (UID: "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:37.738458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.738003    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-envoy\") pod \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") "
Oct 30 03:07:37.738458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.738093    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-kube-api-access-xtzrw\" (UniqueName: \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-kube-api-access-xtzrw\") pod \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") "
Oct 30 03:07:37.738458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.738149    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istiod-ca-cert\") pod \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") "
Oct 30 03:07:37.738458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.738299    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-socket\") pod \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\" (UID: \"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59\") "
Oct 30 03:07:37.739231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.739211    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:37.739271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.739239    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:37.739271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.739253    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:37.739542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.739522    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-socket" pod "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" (UID: "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:37.740658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.740440    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istiod-ca-cert" pod "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" (UID: "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:37.740658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.740519    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-token" pod "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" (UID: "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:37.740845 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.740761    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-podinfo" pod "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" (UID: "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:07:37.740902 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.740879    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-envoy" pod "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" (UID: "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:37.741745 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.741709    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-kube-api-access-xtzrw" pod "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" (UID: "7cfb8f6f-a197-45c9-94ab-4a9d761d0d59"). InnerVolumeSpecName "kube-api-access-xtzrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:37.841414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.841380    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:37.841414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.841414    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtzrw\" (UniqueName: \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-kube-api-access-xtzrw\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:37.841414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.841443    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:37.841414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.841455    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:37.841786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.841468    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:37.841786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:37.841483    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:38.149261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.147687    4278 generic.go:356] "Generic (PLEG): container finished" podID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerID="6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4" exitCode=0
Oct 30 03:07:38.149261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.148112    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" event={"ID":"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59","Type":"ContainerDied","Data":"6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4"}
Oct 30 03:07:38.149261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.148163    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7" event={"ID":"7cfb8f6f-a197-45c9-94ab-4a9d761d0d59","Type":"ContainerDied","Data":"ceccb4ee225e0a2a27138378b9e4e1000d12c9f654f9a09b437324269e2935b8"}
Oct 30 03:07:38.149261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.148186    4278 scope.go:122] "RemoveContainer" containerID="6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4"
Oct 30 03:07:38.149261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.148318    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"
Oct 30 03:07:38.230980 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.230954    4278 scope.go:122] "RemoveContainer" containerID="1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03"
Oct 30 03:07:38.246621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.246581    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"]
Oct 30 03:07:38.249066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.249033    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d7fde0-openshift-default-7ccbfd9d45-6s7v7"]
Oct 30 03:07:38.251481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.250787    4278 scope.go:122] "RemoveContainer" containerID="6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4"
Oct 30 03:07:38.251481 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:07:38.251270    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4\": container with ID starting with 6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4 not found: ID does not exist" containerID="6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4"
Oct 30 03:07:38.251481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.251317    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4"} err="failed to get container status \"6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4\": rpc error: code = NotFound desc = could not find container \"6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4\": container with ID starting with 6601bc5abbbffa3b0b2ee4aeaa620c0d428657c60da0aac16846ddcf8ede94b4 not found: ID does not exist"
Oct 30 03:07:38.251481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.251342    4278 scope.go:122] "RemoveContainer" containerID="1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03"
Oct 30 03:07:38.251836 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:07:38.251815    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03\": container with ID starting with 1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03 not found: ID does not exist" containerID="1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03"
Oct 30 03:07:38.251913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:38.251843    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03"} err="failed to get container status \"1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03\": rpc error: code = NotFound desc = could not find container \"1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03\": container with ID starting with 1f87d3a01b14b4bf346fea96e61757ecebd8ee2c75533dff302b0a3de42fef03 not found: ID does not exist"
Oct 30 03:07:40.201395 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:40.201358    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" path="/var/lib/kubelet/pods/7cfb8f6f-a197-45c9-94ab-4a9d761d0d59/volumes"
Oct 30 03:07:43.464812 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:43.464773    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"]
Oct 30 03:07:43.575557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:43.575521    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:07:48.465821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:48.465760    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896" podUID="d2a0a0b1-f21f-4c43-a509-664da02486a7" containerName="payload-pre-processing" containerID="cri-o://8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5" gracePeriod=25
Oct 30 03:07:48.868390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:48.868352    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:07:48.906853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:48.897467    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a0a0b1-f21f-4c43-a509-664da02486a7-kube-api-access-b854r\" (UniqueName: \"kubernetes.io/projected/d2a0a0b1-f21f-4c43-a509-664da02486a7-kube-api-access-b854r\") pod \"d2a0a0b1-f21f-4c43-a509-664da02486a7\" (UID: \"d2a0a0b1-f21f-4c43-a509-664da02486a7\") "
Oct 30 03:07:48.907415 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:48.907382    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a0a0b1-f21f-4c43-a509-664da02486a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a0a0b1-f21f-4c43-a509-664da02486a7-plugins-config-volume\") pod \"d2a0a0b1-f21f-4c43-a509-664da02486a7\" (UID: \"d2a0a0b1-f21f-4c43-a509-664da02486a7\") "
Oct 30 03:07:48.908135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:48.908106    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a0a0b1-f21f-4c43-a509-664da02486a7-plugins-config-volume" pod "d2a0a0b1-f21f-4c43-a509-664da02486a7" (UID: "d2a0a0b1-f21f-4c43-a509-664da02486a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:48.908582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:48.908539    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a0a0b1-f21f-4c43-a509-664da02486a7-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:48.910637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:48.910613    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a0a0b1-f21f-4c43-a509-664da02486a7-kube-api-access-b854r" pod "d2a0a0b1-f21f-4c43-a509-664da02486a7" (UID: "d2a0a0b1-f21f-4c43-a509-664da02486a7"). InnerVolumeSpecName "kube-api-access-b854r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:49.010908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:49.010872    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b854r\" (UniqueName: \"kubernetes.io/projected/d2a0a0b1-f21f-4c43-a509-664da02486a7-kube-api-access-b854r\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:07:49.360784 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:49.360746    4278 generic.go:356] "Generic (PLEG): container finished" podID="d2a0a0b1-f21f-4c43-a509-664da02486a7" containerID="8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5" exitCode=0
Oct 30 03:07:49.361026 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:49.360789    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"
Oct 30 03:07:49.361026 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:49.360828    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896" event={"ID":"d2a0a0b1-f21f-4c43-a509-664da02486a7","Type":"ContainerDied","Data":"8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5"}
Oct 30 03:07:49.361026 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:49.360904    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896" event={"ID":"d2a0a0b1-f21f-4c43-a509-664da02486a7","Type":"ContainerDied","Data":"19e2d0b3ca20af1683bc7c337bee4455c32007a9182f59ae5b51dd281980a509"}
Oct 30 03:07:49.361026 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:49.361003    4278 scope.go:122] "RemoveContainer" containerID="8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5"
Oct 30 03:07:49.378248 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:49.378226    4278 scope.go:122] "RemoveContainer" containerID="8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5"
Oct 30 03:07:49.378569 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:07:49.378545    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5\": container with ID starting with 8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5 not found: ID does not exist" containerID="8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5"
Oct 30 03:07:49.378621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:49.378581    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5"} err="failed to get container status \"8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5\": rpc error: code = NotFound desc = could not find container \"8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5\": container with ID starting with 8e18d1349e4eec969e1a6a2ff2ee0ada87ca231700b9118f0adc39df1accb8f5 not found: ID does not exist"
Oct 30 03:07:49.416064 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:49.416035    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"]
Oct 30 03:07:49.424975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:49.424945    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1abc2-7795df7847-j9896"]
Oct 30 03:07:50.202008 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:07:50.201966    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a0a0b1-f21f-4c43-a509-664da02486a7" path="/var/lib/kubelet/pods/d2a0a0b1-f21f-4c43-a509-664da02486a7/volumes"
Oct 30 03:07:54.198164 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:07:54.198123    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:08:01.604003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:01.601038    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"]
Oct 30 03:08:01.604003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:01.601468    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy" containerID="cri-o://35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8" gracePeriod=30
Oct 30 03:08:01.874606 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:01.874514    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"]
Oct 30 03:08:01.874770 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:01.874736    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" podUID="c6037223-173b-46fb-87fd-720b538cdf49" containerName="istio-proxy" containerID="cri-o://3c639f4afc4b255bb0aada0c6aca4a550e8c792fc3d49e2f45b82f34e5301f7a" gracePeriod=30
Oct 30 03:08:02.440354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:02.440322    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"]
Oct 30 03:08:02.440555 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:02.440534    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy" containerID="cri-o://028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336" gracePeriod=30
Oct 30 03:08:06.958901 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:06.958874    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 03:08:07.004038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.004006    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:08:07.004191 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.004060    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:08:07.037320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037288    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-socket\") pod \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") "
Oct 30 03:08:07.037525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037399    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-token\" (UniqueName: \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-token\") pod \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") "
Oct 30 03:08:07.037619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037605    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-data\") pod \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") "
Oct 30 03:08:07.037686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037619    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-socket" pod "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" (UID: "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:07.037686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037643    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-podinfo\") pod \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") "
Oct 30 03:08:07.037686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037678    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-credential-socket\") pod \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") "
Oct 30 03:08:07.037844 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037705    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-envoy\") pod \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") "
Oct 30 03:08:07.037844 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037770    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-certs\") pod \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") "
Oct 30 03:08:07.037844 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037827    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-data" pod "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" (UID: "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:07.037986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037901    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istiod-ca-cert\") pod \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") "
Oct 30 03:08:07.037986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037968    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-credential-socket" pod "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" (UID: "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:07.038052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.037985    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-certs" pod "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" (UID: "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:07.038052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.038014    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-kube-api-access-dfn8r\" (UniqueName: \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-kube-api-access-dfn8r\") pod \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\" (UID: \"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a\") "
Oct 30 03:08:07.038266 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.038243    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istiod-ca-cert" pod "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" (UID: "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:07.039396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.039369    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.039568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.039400    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.039568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.039436    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.039568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.039456    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.039568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.039473    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.040126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.040097    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-token" pod "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" (UID: "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:07.040639 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.040618    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-podinfo" pod "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" (UID: "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:08:07.041865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.041840    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-kube-api-access-dfn8r" pod "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" (UID: "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a"). InnerVolumeSpecName "kube-api-access-dfn8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:07.046136 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.046108    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-envoy" pod "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" (UID: "0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:07.142175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.142130    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.142175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.142168    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.142175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.142182    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.142175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.142198    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfn8r\" (UniqueName: \"kubernetes.io/projected/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a-kube-api-access-dfn8r\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.604005 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.603949    4278 generic.go:356] "Generic (PLEG): container finished" podID="c6037223-173b-46fb-87fd-720b538cdf49" containerID="3c639f4afc4b255bb0aada0c6aca4a550e8c792fc3d49e2f45b82f34e5301f7a" exitCode=0
Oct 30 03:08:07.604229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.604190    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" event={"ID":"c6037223-173b-46fb-87fd-720b538cdf49","Type":"ContainerDied","Data":"3c639f4afc4b255bb0aada0c6aca4a550e8c792fc3d49e2f45b82f34e5301f7a"}
Oct 30 03:08:07.619122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.612553    4278 generic.go:356] "Generic (PLEG): container finished" podID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerID="35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8" exitCode=0
Oct 30 03:08:07.619122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.612627    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" event={"ID":"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a","Type":"ContainerDied","Data":"35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8"}
Oct 30 03:08:07.619122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.612671    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z" event={"ID":"0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a","Type":"ContainerDied","Data":"c5232c6dcc8ecbae666e75be49cfcb478e9e57ab4f950d00a880ebe84e537456"}
Oct 30 03:08:07.619122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.612700    4278 scope.go:122] "RemoveContainer" containerID="35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8"
Oct 30 03:08:07.619122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.612864    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"
Oct 30 03:08:07.639542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.634444    4278 scope.go:122] "RemoveContainer" containerID="35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8"
Oct 30 03:08:07.639542 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:08:07.634752    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8\": container with ID starting with 35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8 not found: ID does not exist" containerID="35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8"
Oct 30 03:08:07.639542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.634776    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8"} err="failed to get container status \"35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8\": rpc error: code = NotFound desc = could not find container \"35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8\": container with ID starting with 35775cb8aa7ab4f6280ba4e96fa82c14db4ab0e0d926169f2c853ab5d04c3bc8 not found: ID does not exist"
Oct 30 03:08:07.755000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.754964    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:08:07.876015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.875976    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6037223-173b-46fb-87fd-720b538cdf49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6037223-173b-46fb-87fd-720b538cdf49-istio-podinfo\") pod \"c6037223-173b-46fb-87fd-720b538cdf49\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") "
Oct 30 03:08:07.876186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.876088    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-socket\") pod \"c6037223-173b-46fb-87fd-720b538cdf49\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") "
Oct 30 03:08:07.876186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.876124    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6037223-173b-46fb-87fd-720b538cdf49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6037223-173b-46fb-87fd-720b538cdf49-istiod-ca-cert\") pod \"c6037223-173b-46fb-87fd-720b538cdf49\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") "
Oct 30 03:08:07.876325 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.876247    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-istio-token\" (UniqueName: \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-istio-token\") pod \"c6037223-173b-46fb-87fd-720b538cdf49\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") "
Oct 30 03:08:07.876376 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.876339    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-certs\") pod \"c6037223-173b-46fb-87fd-720b538cdf49\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") "
Oct 30 03:08:07.876444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.876398    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-data\") pod \"c6037223-173b-46fb-87fd-720b538cdf49\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") "
Oct 30 03:08:07.876505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.876477    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-kube-api-access-6skkr\" (UniqueName: \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-kube-api-access-6skkr\") pod \"c6037223-173b-46fb-87fd-720b538cdf49\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") "
Oct 30 03:08:07.876626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.876606    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-envoy\") pod \"c6037223-173b-46fb-87fd-720b538cdf49\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") "
Oct 30 03:08:07.876721 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:07.876704    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-credential-socket\") pod \"c6037223-173b-46fb-87fd-720b538cdf49\" (UID: \"c6037223-173b-46fb-87fd-720b538cdf49\") "
Oct 30 03:08:08.016653 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.016238    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-credential-socket" pod "c6037223-173b-46fb-87fd-720b538cdf49" (UID: "c6037223-173b-46fb-87fd-720b538cdf49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:08.021764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.021191    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-certs" pod "c6037223-173b-46fb-87fd-720b538cdf49" (UID: "c6037223-173b-46fb-87fd-720b538cdf49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:08.021764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.021395    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-data" pod "c6037223-173b-46fb-87fd-720b538cdf49" (UID: "c6037223-173b-46fb-87fd-720b538cdf49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:08.033288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.030179    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-socket" pod "c6037223-173b-46fb-87fd-720b538cdf49" (UID: "c6037223-173b-46fb-87fd-720b538cdf49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:08.033288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.031080    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6037223-173b-46fb-87fd-720b538cdf49-istiod-ca-cert" pod "c6037223-173b-46fb-87fd-720b538cdf49" (UID: "c6037223-173b-46fb-87fd-720b538cdf49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:08.033288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.033209    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"]
Oct 30 03:08:08.052559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.052516    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-envoy" pod "c6037223-173b-46fb-87fd-720b538cdf49" (UID: "c6037223-173b-46fb-87fd-720b538cdf49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:08.058064 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.058020    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-istio-token" pod "c6037223-173b-46fb-87fd-720b538cdf49" (UID: "c6037223-173b-46fb-87fd-720b538cdf49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:08.063638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.063595    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.063811 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.063653    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.063811 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.063669    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.063811 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.063683    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.063811 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.063703    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.063811 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.063716    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6037223-173b-46fb-87fd-720b538cdf49-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.063811 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.063732    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6037223-173b-46fb-87fd-720b538cdf49-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.067442 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.067399    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-bhs4z"]
Oct 30 03:08:08.067849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.067818    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6037223-173b-46fb-87fd-720b538cdf49-istio-podinfo" pod "c6037223-173b-46fb-87fd-720b538cdf49" (UID: "c6037223-173b-46fb-87fd-720b538cdf49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:08:08.074132 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.073927    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-kube-api-access-6skkr" pod "c6037223-173b-46fb-87fd-720b538cdf49" (UID: "c6037223-173b-46fb-87fd-720b538cdf49"). InnerVolumeSpecName "kube-api-access-6skkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:08.085089 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.084786    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 03:08:08.165109 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.165069    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-data\") pod \"36064a86-b94d-478a-aa06-bf99360bff43\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") "
Oct 30 03:08:08.165285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.165198    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-envoy\") pod \"36064a86-b94d-478a-aa06-bf99360bff43\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") "
Oct 30 03:08:08.165285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.165264    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36064a86-b94d-478a-aa06-bf99360bff43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36064a86-b94d-478a-aa06-bf99360bff43-istio-podinfo\") pod \"36064a86-b94d-478a-aa06-bf99360bff43\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") "
Oct 30 03:08:08.165357 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.165344    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-credential-socket\") pod \"36064a86-b94d-478a-aa06-bf99360bff43\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") "
Oct 30 03:08:08.165418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.165398    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-istio-token\" (UniqueName: \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-istio-token\") pod \"36064a86-b94d-478a-aa06-bf99360bff43\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") "
Oct 30 03:08:08.165599 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.165519    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-socket\") pod \"36064a86-b94d-478a-aa06-bf99360bff43\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") "
Oct 30 03:08:08.165599 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.165546    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36064a86-b94d-478a-aa06-bf99360bff43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36064a86-b94d-478a-aa06-bf99360bff43-istiod-ca-cert\") pod \"36064a86-b94d-478a-aa06-bf99360bff43\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") "
Oct 30 03:08:08.165694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.165635    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-certs\") pod \"36064a86-b94d-478a-aa06-bf99360bff43\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") "
Oct 30 03:08:08.165733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.165697    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-kube-api-access-ckxxk\" (UniqueName: \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-kube-api-access-ckxxk\") pod \"36064a86-b94d-478a-aa06-bf99360bff43\" (UID: \"36064a86-b94d-478a-aa06-bf99360bff43\") "
Oct 30 03:08:08.166490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.166468    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6skkr\" (UniqueName: \"kubernetes.io/projected/c6037223-173b-46fb-87fd-720b538cdf49-kube-api-access-6skkr\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.166544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.166497    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6037223-173b-46fb-87fd-720b538cdf49-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.167019 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.166995    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-credential-socket" pod "36064a86-b94d-478a-aa06-bf99360bff43" (UID: "36064a86-b94d-478a-aa06-bf99360bff43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:08.167210 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.167185    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-data" pod "36064a86-b94d-478a-aa06-bf99360bff43" (UID: "36064a86-b94d-478a-aa06-bf99360bff43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:08.168727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.168670    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-socket" pod "36064a86-b94d-478a-aa06-bf99360bff43" (UID: "36064a86-b94d-478a-aa06-bf99360bff43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:08.168890 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.168864    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-certs" pod "36064a86-b94d-478a-aa06-bf99360bff43" (UID: "36064a86-b94d-478a-aa06-bf99360bff43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:08.169003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.168925    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-kube-api-access-ckxxk" pod "36064a86-b94d-478a-aa06-bf99360bff43" (UID: "36064a86-b94d-478a-aa06-bf99360bff43"). InnerVolumeSpecName "kube-api-access-ckxxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:08.169080 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.169059    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36064a86-b94d-478a-aa06-bf99360bff43-istiod-ca-cert" pod "36064a86-b94d-478a-aa06-bf99360bff43" (UID: "36064a86-b94d-478a-aa06-bf99360bff43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:08.169307 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.169277    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-envoy" pod "36064a86-b94d-478a-aa06-bf99360bff43" (UID: "36064a86-b94d-478a-aa06-bf99360bff43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:08.170937 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.170524    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-istio-token" pod "36064a86-b94d-478a-aa06-bf99360bff43" (UID: "36064a86-b94d-478a-aa06-bf99360bff43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:08.170937 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.170855    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36064a86-b94d-478a-aa06-bf99360bff43-istio-podinfo" pod "36064a86-b94d-478a-aa06-bf99360bff43" (UID: "36064a86-b94d-478a-aa06-bf99360bff43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:08:08.203240 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.203202    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" path="/var/lib/kubelet/pods/0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a/volumes"
Oct 30 03:08:08.267817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.267744    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36064a86-b94d-478a-aa06-bf99360bff43-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.267817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.267771    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.267817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.267781    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.267817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.267789    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.267817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.267799    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36064a86-b94d-478a-aa06-bf99360bff43-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.267817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.267807    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.267817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.267815    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckxxk\" (UniqueName: \"kubernetes.io/projected/36064a86-b94d-478a-aa06-bf99360bff43-kube-api-access-ckxxk\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.268128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.267825    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.268128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.267833    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36064a86-b94d-478a-aa06-bf99360bff43-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:08.622262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.622177    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"
Oct 30 03:08:08.622262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.622204    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg" event={"ID":"c6037223-173b-46fb-87fd-720b538cdf49","Type":"ContainerDied","Data":"749ccfcafc62b1a6b7f2a7b6c073beaabdb900b7d191d13067aa1737be25b0d4"}
Oct 30 03:08:08.622551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.622271    4278 scope.go:122] "RemoveContainer" containerID="3c639f4afc4b255bb0aada0c6aca4a550e8c792fc3d49e2f45b82f34e5301f7a"
Oct 30 03:08:08.625864 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.625835    4278 generic.go:356] "Generic (PLEG): container finished" podID="36064a86-b94d-478a-aa06-bf99360bff43" containerID="028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336" exitCode=0
Oct 30 03:08:08.625991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.625876    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" event={"ID":"36064a86-b94d-478a-aa06-bf99360bff43","Type":"ContainerDied","Data":"028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336"}
Oct 30 03:08:08.625991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.625935    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm" event={"ID":"36064a86-b94d-478a-aa06-bf99360bff43","Type":"ContainerDied","Data":"dc78520dc1dd557063f1d840e87d0df653f89b9e59967b3aa84584049ab087ae"}
Oct 30 03:08:08.625991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.625891    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"
Oct 30 03:08:08.635870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.635846    4278 scope.go:122] "RemoveContainer" containerID="028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336"
Oct 30 03:08:08.647077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.647057    4278 scope.go:122] "RemoveContainer" containerID="028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336"
Oct 30 03:08:08.647377 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:08:08.647349    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336\": container with ID starting with 028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336 not found: ID does not exist" containerID="028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336"
Oct 30 03:08:08.647457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.647379    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336"} err="failed to get container status \"028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336\": rpc error: code = NotFound desc = could not find container \"028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336\": container with ID starting with 028d5c6df954d2875e3363c917e789d8b8a05b682b2f67cf5b6ba70c0d45f336 not found: ID does not exist"
Oct 30 03:08:08.700305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.700278    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"]
Oct 30 03:08:08.713292 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.713264    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-zn4gg"]
Oct 30 03:08:08.742016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.741992    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"]
Oct 30 03:08:08.753217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:08.753196    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5q5lm"]
Oct 30 03:08:09.573660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.573629    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"]
Oct 30 03:08:09.574284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574270    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy"
Oct 30 03:08:09.574334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574285    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy"
Oct 30 03:08:09.574334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574304    4278 cpu_manager.go:414] "containerMap: removing container" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy"
Oct 30 03:08:09.574334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574309    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy"
Oct 30 03:08:09.574334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574319    4278 cpu_manager.go:414] "containerMap: removing container" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy"
Oct 30 03:08:09.574334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574323    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy"
Oct 30 03:08:09.574334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574333    4278 cpu_manager.go:414] "containerMap: removing container" podUID="d2a0a0b1-f21f-4c43-a509-664da02486a7" containerName="payload-pre-processing"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574338    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a0a0b1-f21f-4c43-a509-664da02486a7" containerName="payload-pre-processing"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574352    4278 cpu_manager.go:414] "containerMap: removing container" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574356    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574373    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a61bfe15-f42d-424a-9376-37ec7d48581d" containerName="revoke-keys"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574378    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61bfe15-f42d-424a-9376-37ec7d48581d" containerName="revoke-keys"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574393    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c6037223-173b-46fb-87fd-720b538cdf49" containerName="istio-proxy"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574398    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6037223-173b-46fb-87fd-720b538cdf49" containerName="istio-proxy"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574503    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="36064a86-b94d-478a-aa06-bf99360bff43" containerName="istio-proxy"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574510    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574517    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a0a0b1-f21f-4c43-a509-664da02486a7" containerName="payload-pre-processing"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574526    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61bfe15-f42d-424a-9376-37ec7d48581d" containerName="revoke-keys"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574534    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6037223-173b-46fb-87fd-720b538cdf49" containerName="istio-proxy"
Oct 30 03:08:09.574539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.574542    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1de8f8-8a94-4f5e-b74e-fcce4ccb243a" containerName="istio-proxy"
Oct 30 03:08:09.579782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.579759    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.598125 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.598104    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-be9f2a59-openshift-default-dockercfg-4gctp\""
Oct 30 03:08:09.616768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.616744    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"]
Oct 30 03:08:09.689850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.689815    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.690014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.689928    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfgc\" (UniqueName: \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-kube-api-access-brfgc\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.690014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.689972    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.690228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.690203    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.690371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.690350    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.690455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.690415    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.690527 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.690511    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.690600 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.690586    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.690661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.690632    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.792453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.792404    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.792607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.792534    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.792607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.792585    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.792730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.792704    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.792824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.792790    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.792890 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.792842    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.792950 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.792895    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.793006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.792955    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfgc\" (UniqueName: \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-kube-api-access-brfgc\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.793006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.793001    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.793489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.793176    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.793489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.793235    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.793489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.793296    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.793489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.793331    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.793776 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.793541    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.795248 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.795223    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.795608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.795589    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.813841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.813802    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.813990 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.813974    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfgc\" (UniqueName: \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-kube-api-access-brfgc\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:09.897551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:09.897474    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:10.205182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:10.205095    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36064a86-b94d-478a-aa06-bf99360bff43" path="/var/lib/kubelet/pods/36064a86-b94d-478a-aa06-bf99360bff43/volumes"
Oct 30 03:08:10.205661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:10.205640    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6037223-173b-46fb-87fd-720b538cdf49" path="/var/lib/kubelet/pods/c6037223-173b-46fb-87fd-720b538cdf49/volumes"
Oct 30 03:08:10.268119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:10.268087    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"]
Oct 30 03:08:10.268249 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:08:10.268186    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc6b7b8_98a4_46ca_b4d8_710f8a576a02.slice/crio-c02931dad31bc596f4a0934be88a757d7074bf4e1fcb78002f61000ad8fe7075 WatchSource:0}: Error finding container c02931dad31bc596f4a0934be88a757d7074bf4e1fcb78002f61000ad8fe7075: Status 404 returned error can't find the container with id c02931dad31bc596f4a0934be88a757d7074bf4e1fcb78002f61000ad8fe7075
Oct 30 03:08:10.272307 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:10.272270    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:10.272438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:10.272356    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:10.272494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:10.272444    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:10.646129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:10.646091    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" event={"ID":"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02","Type":"ContainerStarted","Data":"c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093"}
Oct 30 03:08:10.646610 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:10.646147    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" event={"ID":"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02","Type":"ContainerStarted","Data":"c02931dad31bc596f4a0934be88a757d7074bf4e1fcb78002f61000ad8fe7075"}
Oct 30 03:08:10.898489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:10.898404    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:11.899543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:11.899496    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:08:11.899940 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:11.899571    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 03:08:12.913015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:12.912539    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:08:12.913015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:12.912614    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 03:08:13.901507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:13.898589    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:08:13.901507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:13.898666    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 03:08:14.898214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:14.898170    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:08:14.898838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:14.898252    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 03:08:15.898478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:15.898408    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:08:15.898865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:15.898517    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 03:08:16.385565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.385492    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" podStartSLOduration=7.385418479 podStartE2EDuration="7.385418479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:10.715970682 +0000 UTC m=+1628.640178870" watchObservedRunningTime="2026-10-30 03:08:16.385418479 +0000 UTC m=+1634.309626667"
Oct 30 03:08:16.389883 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.389856    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8809-dfdaee"]
Oct 30 03:08:16.390689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.390672    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfb8f6f-a197-45c9-94ab-4a9d761d0d59" containerName="istio-proxy"
Oct 30 03:08:16.394625 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.394609    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-dfdaee"
Oct 30 03:08:16.409082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.409052    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8809-dfdaee"]
Oct 30 03:08:16.417819 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.417791    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7q6n5\""
Oct 30 03:08:16.500954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.500917    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwmx7\" (UniqueName: \"kubernetes.io/projected/d6ee08ab-6a61-425e-a5d4-9b99c077a38b-kube-api-access-cwmx7\") pod \"test-curl-8809-dfdaee\" (UID: \"d6ee08ab-6a61-425e-a5d4-9b99c077a38b\") " pod="opendatahub/test-curl-8809-dfdaee"
Oct 30 03:08:16.602998 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.602961    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwmx7\" (UniqueName: \"kubernetes.io/projected/d6ee08ab-6a61-425e-a5d4-9b99c077a38b-kube-api-access-cwmx7\") pod \"test-curl-8809-dfdaee\" (UID: \"d6ee08ab-6a61-425e-a5d4-9b99c077a38b\") " pod="opendatahub/test-curl-8809-dfdaee"
Oct 30 03:08:16.613191 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.613160    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"]
Oct 30 03:08:16.613364 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.613343    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy" containerID="cri-o://eff671dc579ecf252b1f9440ca427baf353415b198c2e73bf9f4ca298b6a911b" gracePeriod=30
Oct 30 03:08:16.634660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.634621    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwmx7\" (UniqueName: \"kubernetes.io/projected/d6ee08ab-6a61-425e-a5d4-9b99c077a38b-kube-api-access-cwmx7\") pod \"test-curl-8809-dfdaee\" (UID: \"d6ee08ab-6a61-425e-a5d4-9b99c077a38b\") " pod="opendatahub/test-curl-8809-dfdaee"
Oct 30 03:08:16.709311 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.709232    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-dfdaee"
Oct 30 03:08:16.900035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.898223    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:08:16.900035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:16.898268    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 03:08:17.073457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:17.073413    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8809-dfdaee"]
Oct 30 03:08:17.076794 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:08:17.076758    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6ee08ab_6a61_425e_a5d4_9b99c077a38b.slice/crio-897ee3d73e5e218730b3c76b702ddd7f52e96f18d149af373a377f2b5e300fd3 WatchSource:0}: Error finding container 897ee3d73e5e218730b3c76b702ddd7f52e96f18d149af373a377f2b5e300fd3: Status 404 returned error can't find the container with id 897ee3d73e5e218730b3c76b702ddd7f52e96f18d149af373a377f2b5e300fd3
Oct 30 03:08:17.830812 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:17.830772    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8809-dfdaee" event={"ID":"d6ee08ab-6a61-425e-a5d4-9b99c077a38b","Type":"ContainerStarted","Data":"215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0"}
Oct 30 03:08:17.830812 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:17.830819    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8809-dfdaee" event={"ID":"d6ee08ab-6a61-425e-a5d4-9b99c077a38b","Type":"ContainerStarted","Data":"897ee3d73e5e218730b3c76b702ddd7f52e96f18d149af373a377f2b5e300fd3"}
Oct 30 03:08:17.873554 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:17.873494    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8809-dfdaee" podStartSLOduration=1.873477935 podStartE2EDuration="1.873477935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:17.868276666 +0000 UTC m=+1635.792484854" watchObservedRunningTime="2026-10-30 03:08:17.873477935 +0000 UTC m=+1635.797686123"
Oct 30 03:08:17.903414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:17.903381    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:17.903904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:17.903881    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:17.904706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:17.904685    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:08:18.802015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:18.801977    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8809-dfdaee"]
Oct 30 03:08:18.810988 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:18.810956    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8809-dfdaee"]
Oct 30 03:08:19.058514 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.058362    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"]
Oct 30 03:08:19.059129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.059108    4278 cpu_manager.go:414] "containerMap: removing container" podUID="d6ee08ab-6a61-425e-a5d4-9b99c077a38b" containerName="test-curl-8809-dfdaee"
Oct 30 03:08:19.059129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.059129    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ee08ab-6a61-425e-a5d4-9b99c077a38b" containerName="test-curl-8809-dfdaee"
Oct 30 03:08:19.059279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.059269    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ee08ab-6a61-425e-a5d4-9b99c077a38b" containerName="test-curl-8809-dfdaee"
Oct 30 03:08:19.063415 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.063393    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.083750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.083717    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"]
Oct 30 03:08:19.088505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.088479    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:08:19.088624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.088517    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:08:19.154635 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.154594    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2gwq\" (UniqueName: \"kubernetes.io/projected/8a527b68-095f-4b83-abbc-df94b88dff35-kube-api-access-l2gwq\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.154846 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.154821    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.154913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.154873    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.255809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.255774    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2gwq\" (UniqueName: \"kubernetes.io/projected/8a527b68-095f-4b83-abbc-df94b88dff35-kube-api-access-l2gwq\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.256023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.255970    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.256023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.256013    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.256900 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.256876    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.258620 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.258598    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.275885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.275850    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2gwq\" (UniqueName: \"kubernetes.io/projected/8a527b68-095f-4b83-abbc-df94b88dff35-kube-api-access-l2gwq\") pod \"maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.379643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.379548    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:19.705463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.705356    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"]
Oct 30 03:08:19.709071 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:08:19.709036    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a527b68_095f_4b83_abbc_df94b88dff35.slice/crio-f1d312eb50209d338358d2ea1a226f33c4145af0d647ba726b8f9fb3565af5cb WatchSource:0}: Error finding container f1d312eb50209d338358d2ea1a226f33c4145af0d647ba726b8f9fb3565af5cb: Status 404 returned error can't find the container with id f1d312eb50209d338358d2ea1a226f33c4145af0d647ba726b8f9fb3565af5cb
Oct 30 03:08:19.849138 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.849096    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2" event={"ID":"8a527b68-095f-4b83-abbc-df94b88dff35","Type":"ContainerStarted","Data":"f1d312eb50209d338358d2ea1a226f33c4145af0d647ba726b8f9fb3565af5cb"}
Oct 30 03:08:19.849320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:19.849222    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8809-dfdaee" podUID="d6ee08ab-6a61-425e-a5d4-9b99c077a38b" containerName="test-curl-8809-dfdaee" containerID="cri-o://215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0" gracePeriod=2
Oct 30 03:08:19.862530 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:08:19.862492    4278 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8809-dfdaee\" is forbidden: User \"system:node:ip-10-0-128-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-214.ec2.internal' and this object" podUID="d6ee08ab-6a61-425e-a5d4-9b99c077a38b" pod="opendatahub/test-curl-8809-dfdaee"
Oct 30 03:08:20.859381 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:20.859280    4278 generic.go:356] "Generic (PLEG): container finished" podID="8a527b68-095f-4b83-abbc-df94b88dff35" containerID="40baaef43c41bb248d90933775bc5c95af8aa75b3d6996c58c048bc3bc5ad2e7" exitCode=0
Oct 30 03:08:20.859381 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:20.859363    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2" event={"ID":"8a527b68-095f-4b83-abbc-df94b88dff35","Type":"ContainerDied","Data":"40baaef43c41bb248d90933775bc5c95af8aa75b3d6996c58c048bc3bc5ad2e7"}
Oct 30 03:08:21.874310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:21.874276    4278 generic.go:356] "Generic (PLEG): container finished" podID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerID="eff671dc579ecf252b1f9440ca427baf353415b198c2e73bf9f4ca298b6a911b" exitCode=0
Oct 30 03:08:21.874727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:21.874335    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" event={"ID":"44332434-cfc7-4809-a3f5-4cb038a2a794","Type":"ContainerDied","Data":"eff671dc579ecf252b1f9440ca427baf353415b198c2e73bf9f4ca298b6a911b"}
Oct 30 03:08:21.943602 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:21.942917    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 03:08:22.002118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.001375    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44332434-cfc7-4809-a3f5-4cb038a2a794-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44332434-cfc7-4809-a3f5-4cb038a2a794-istiod-ca-cert\") pod \"44332434-cfc7-4809-a3f5-4cb038a2a794\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") "
Oct 30 03:08:22.002118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.001475    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-podinfo\") pod \"44332434-cfc7-4809-a3f5-4cb038a2a794\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") "
Oct 30 03:08:22.002118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.001576    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-credential-socket\") pod \"44332434-cfc7-4809-a3f5-4cb038a2a794\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") "
Oct 30 03:08:22.002118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.001692    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-data\") pod \"44332434-cfc7-4809-a3f5-4cb038a2a794\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") "
Oct 30 03:08:22.002118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.001809    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-token\" (UniqueName: \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-token\") pod \"44332434-cfc7-4809-a3f5-4cb038a2a794\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") "
Oct 30 03:08:22.002118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.001923    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-certs\") pod \"44332434-cfc7-4809-a3f5-4cb038a2a794\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") "
Oct 30 03:08:22.002118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.002002    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-kube-api-access-wndrz\" (UniqueName: \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-kube-api-access-wndrz\") pod \"44332434-cfc7-4809-a3f5-4cb038a2a794\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") "
Oct 30 03:08:22.002118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.002073    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-socket\") pod \"44332434-cfc7-4809-a3f5-4cb038a2a794\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") "
Oct 30 03:08:22.002711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.002639    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-certs" pod "44332434-cfc7-4809-a3f5-4cb038a2a794" (UID: "44332434-cfc7-4809-a3f5-4cb038a2a794"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:22.002711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.002653    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-credential-socket" pod "44332434-cfc7-4809-a3f5-4cb038a2a794" (UID: "44332434-cfc7-4809-a3f5-4cb038a2a794"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:22.002897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.002870    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-data" pod "44332434-cfc7-4809-a3f5-4cb038a2a794" (UID: "44332434-cfc7-4809-a3f5-4cb038a2a794"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:22.003766 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.003717    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-socket" pod "44332434-cfc7-4809-a3f5-4cb038a2a794" (UID: "44332434-cfc7-4809-a3f5-4cb038a2a794"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:22.007995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.002124    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-envoy\") pod \"44332434-cfc7-4809-a3f5-4cb038a2a794\" (UID: \"44332434-cfc7-4809-a3f5-4cb038a2a794\") "
Oct 30 03:08:22.007995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.007232    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.007995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.007264    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.007995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.007282    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.007995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.007296    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.008815 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.008695    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-kube-api-access-wndrz" pod "44332434-cfc7-4809-a3f5-4cb038a2a794" (UID: "44332434-cfc7-4809-a3f5-4cb038a2a794"). InnerVolumeSpecName "kube-api-access-wndrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:22.011148 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.010649    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-podinfo" pod "44332434-cfc7-4809-a3f5-4cb038a2a794" (UID: "44332434-cfc7-4809-a3f5-4cb038a2a794"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:08:22.011400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.011179    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-envoy" pod "44332434-cfc7-4809-a3f5-4cb038a2a794" (UID: "44332434-cfc7-4809-a3f5-4cb038a2a794"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:22.011400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.011268    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-token" pod "44332434-cfc7-4809-a3f5-4cb038a2a794" (UID: "44332434-cfc7-4809-a3f5-4cb038a2a794"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:22.011906 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.011860    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44332434-cfc7-4809-a3f5-4cb038a2a794-istiod-ca-cert" pod "44332434-cfc7-4809-a3f5-4cb038a2a794" (UID: "44332434-cfc7-4809-a3f5-4cb038a2a794"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:22.075914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.075873    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:22.108329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.108294    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-openshift-ca\") pod \"8a527b68-095f-4b83-abbc-df94b88dff35\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") "
Oct 30 03:08:22.108599 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.108581    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a527b68-095f-4b83-abbc-df94b88dff35-kube-api-access-l2gwq\" (UniqueName: \"kubernetes.io/projected/8a527b68-095f-4b83-abbc-df94b88dff35-kube-api-access-l2gwq\") pod \"8a527b68-095f-4b83-abbc-df94b88dff35\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") "
Oct 30 03:08:22.108653 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.108629    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-xks-ca\") pod \"8a527b68-095f-4b83-abbc-df94b88dff35\" (UID: \"8a527b68-095f-4b83-abbc-df94b88dff35\") "
Oct 30 03:08:22.108724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.108694    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-openshift-ca" pod "8a527b68-095f-4b83-abbc-df94b88dff35" (UID: "8a527b68-095f-4b83-abbc-df94b88dff35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:22.109497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.109471    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.109497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.109497    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44332434-cfc7-4809-a3f5-4cb038a2a794-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.109497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.109507    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.109758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.109516    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.109758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.109526    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.109758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.109535    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wndrz\" (UniqueName: \"kubernetes.io/projected/44332434-cfc7-4809-a3f5-4cb038a2a794-kube-api-access-wndrz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.110848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.110817    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-xks-ca" pod "8a527b68-095f-4b83-abbc-df94b88dff35" (UID: "8a527b68-095f-4b83-abbc-df94b88dff35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:08:22.111012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.110992    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a527b68-095f-4b83-abbc-df94b88dff35-kube-api-access-l2gwq" pod "8a527b68-095f-4b83-abbc-df94b88dff35" (UID: "8a527b68-095f-4b83-abbc-df94b88dff35"). InnerVolumeSpecName "kube-api-access-l2gwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:22.122294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.122265    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-dfdaee"
Oct 30 03:08:22.210079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.209978    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ee08ab-6a61-425e-a5d4-9b99c077a38b-kube-api-access-cwmx7\" (UniqueName: \"kubernetes.io/projected/d6ee08ab-6a61-425e-a5d4-9b99c077a38b-kube-api-access-cwmx7\") pod \"d6ee08ab-6a61-425e-a5d4-9b99c077a38b\" (UID: \"d6ee08ab-6a61-425e-a5d4-9b99c077a38b\") "
Oct 30 03:08:22.210719 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.210695    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2gwq\" (UniqueName: \"kubernetes.io/projected/8a527b68-095f-4b83-abbc-df94b88dff35-kube-api-access-l2gwq\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.210833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.210722    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a527b68-095f-4b83-abbc-df94b88dff35-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.212293 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.212265    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ee08ab-6a61-425e-a5d4-9b99c077a38b-kube-api-access-cwmx7" pod "d6ee08ab-6a61-425e-a5d4-9b99c077a38b" (UID: "d6ee08ab-6a61-425e-a5d4-9b99c077a38b"). InnerVolumeSpecName "kube-api-access-cwmx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:22.312901 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.312866    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwmx7\" (UniqueName: \"kubernetes.io/projected/d6ee08ab-6a61-425e-a5d4-9b99c077a38b-kube-api-access-cwmx7\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:22.883969 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.883932    4278 generic.go:356] "Generic (PLEG): container finished" podID="d6ee08ab-6a61-425e-a5d4-9b99c077a38b" containerID="215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0" exitCode=137
Oct 30 03:08:22.884529 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.883983    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-dfdaee"
Oct 30 03:08:22.884529 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.884042    4278 scope.go:122] "RemoveContainer" containerID="215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0"
Oct 30 03:08:22.885855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.885835    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"
Oct 30 03:08:22.885984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.885861    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2" event={"ID":"8a527b68-095f-4b83-abbc-df94b88dff35","Type":"ContainerDied","Data":"f1d312eb50209d338358d2ea1a226f33c4145af0d647ba726b8f9fb3565af5cb"}
Oct 30 03:08:22.885984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.885917    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1d312eb50209d338358d2ea1a226f33c4145af0d647ba726b8f9fb3565af5cb"
Oct 30 03:08:22.887665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.887637    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m" event={"ID":"44332434-cfc7-4809-a3f5-4cb038a2a794","Type":"ContainerDied","Data":"dd0979b75018525380ed006a627a68ba2842a8f954a544a22e3e05372ce7f4b2"}
Oct 30 03:08:22.887665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.887665    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"
Oct 30 03:08:22.896913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.896888    4278 scope.go:122] "RemoveContainer" containerID="215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0"
Oct 30 03:08:22.897262 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:08:22.897242    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0\": container with ID starting with 215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0 not found: ID does not exist" containerID="215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0"
Oct 30 03:08:22.897310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.897270    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0"} err="failed to get container status \"215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0\": rpc error: code = NotFound desc = could not find container \"215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0\": container with ID starting with 215357e60e701ec5d9dffb2483b2fdb53fd60b518b877f6a0c93e8c409956ea0 not found: ID does not exist"
Oct 30 03:08:22.897310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.897290    4278 scope.go:122] "RemoveContainer" containerID="eff671dc579ecf252b1f9440ca427baf353415b198c2e73bf9f4ca298b6a911b"
Oct 30 03:08:22.954617 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.954575    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"]
Oct 30 03:08:22.965318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:22.965277    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8547m"]
Oct 30 03:08:24.203868 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:08:24.203827    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:08:24.206345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.206314    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" path="/var/lib/kubelet/pods/44332434-cfc7-4809-a3f5-4cb038a2a794/volumes"
Oct 30 03:08:24.207009 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.206986    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ee08ab-6a61-425e-a5d4-9b99c077a38b" path="/var/lib/kubelet/pods/d6ee08ab-6a61-425e-a5d4-9b99c077a38b/volumes"
Oct 30 03:08:24.677739 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.677700    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"]
Oct 30 03:08:24.678464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.678447    4278 cpu_manager.go:414] "containerMap: removing container" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy"
Oct 30 03:08:24.678464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.678464    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy"
Oct 30 03:08:24.678593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.678493    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8a527b68-095f-4b83-abbc-df94b88dff35" containerName="revoke-keys"
Oct 30 03:08:24.678593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.678499    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a527b68-095f-4b83-abbc-df94b88dff35" containerName="revoke-keys"
Oct 30 03:08:24.678665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.678617    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="44332434-cfc7-4809-a3f5-4cb038a2a794" containerName="istio-proxy"
Oct 30 03:08:24.678665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.678629    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a527b68-095f-4b83-abbc-df94b88dff35" containerName="revoke-keys"
Oct 30 03:08:24.682569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.682548    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.720066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.720027    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"]
Oct 30 03:08:24.762396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.762356    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq26d\" (UniqueName: \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-kube-api-access-qq26d\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.762608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.762451    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.762608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.762564    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7588026-2174-4904-a2af-8e4cdf2873f5-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.762776 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.762746    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.762841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.762819    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.763013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.762993    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.763051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.763034    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.763098 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.763075    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.763164 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.763141    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.873436 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.873085    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.873436 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.873240    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq26d\" (UniqueName: \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-kube-api-access-qq26d\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.873436 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.873305    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.874452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.873777    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7588026-2174-4904-a2af-8e4cdf2873f5-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.874452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.873970    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.874452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.874040    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.874452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.874232    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.874452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.874269    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.874452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.874312    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.876350 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.875314    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.876350 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.875612    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.876878 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.876852    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.877178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.877149    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7588026-2174-4904-a2af-8e4cdf2873f5-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.877524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.877495    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.879173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.879141    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.879916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.879895    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.898355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.898330    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq26d\" (UniqueName: \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-kube-api-access-qq26d\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:24.898506 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:24.898328    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:25.002966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:25.002865    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:25.353008 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:25.352959    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"]
Oct 30 03:08:25.355865 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:08:25.355837    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7588026_2174_4904_a2af_8e4cdf2873f5.slice/crio-4d416a12c38f1cdca9db177cf1dba2babd823d595318c28c0ee32469a9292f87 WatchSource:0}: Error finding container 4d416a12c38f1cdca9db177cf1dba2babd823d595318c28c0ee32469a9292f87: Status 404 returned error can't find the container with id 4d416a12c38f1cdca9db177cf1dba2babd823d595318c28c0ee32469a9292f87
Oct 30 03:08:25.358999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:25.358966    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:25.359050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:25.359042    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:25.359099 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:25.359085    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:25.948239 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:25.948197    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" event={"ID":"c7588026-2174-4904-a2af-8e4cdf2873f5","Type":"ContainerStarted","Data":"e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394"}
Oct 30 03:08:25.948469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:25.948268    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" event={"ID":"c7588026-2174-4904-a2af-8e4cdf2873f5","Type":"ContainerStarted","Data":"4d416a12c38f1cdca9db177cf1dba2babd823d595318c28c0ee32469a9292f87"}
Oct 30 03:08:26.003342 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:26.003297    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:26.005018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:26.004993    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 03:08:26.005164 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:26.005042    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 03:08:26.041399 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:26.041332    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" podStartSLOduration=2.041310788 podStartE2EDuration="2.041310788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:26.036642145 +0000 UTC m=+1643.960850334" watchObservedRunningTime="2026-10-30 03:08:26.041310788 +0000 UTC m=+1643.965518978"
Oct 30 03:08:27.005654 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:27.005611    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 03:08:27.006220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:27.005671    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 03:08:28.003864 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:28.003824    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 03:08:28.004104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:28.003889    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 03:08:29.003711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:29.003678    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 03:08:29.004095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:29.003760    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 03:08:30.003949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.003904    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 03:08:30.004417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.003987    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 03:08:30.145193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.145154    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"]
Oct 30 03:08:30.150181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.150161    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.189452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.183674    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:08:30.189452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.183809    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:08:30.223726 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.223683    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"]
Oct 30 03:08:30.240877 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.240826    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbj2q\" (UniqueName: \"kubernetes.io/projected/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-kube-api-access-hbj2q\") pod \"maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.241046 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.240969    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.241329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.241306    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.354449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.351361    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.354449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.351771    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbj2q\" (UniqueName: \"kubernetes.io/projected/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-kube-api-access-hbj2q\") pod \"maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.377449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.364654    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.377449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.366114    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.389458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.380029    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.399493 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.399212    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbj2q\" (UniqueName: \"kubernetes.io/projected/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-kube-api-access-hbj2q\") pod \"maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.510449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.507750    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:30.999213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:30.999173    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"]
Oct 30 03:08:30.999394 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:08:30.999127    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e0df2d_3fd3_414c_8fc0_b21092f06b1f.slice/crio-f464381ec540929e512c60035e1e1496bf5c7af75499dd3380e3d0a5d446bc5c WatchSource:0}: Error finding container f464381ec540929e512c60035e1e1496bf5c7af75499dd3380e3d0a5d446bc5c: Status 404 returned error can't find the container with id f464381ec540929e512c60035e1e1496bf5c7af75499dd3380e3d0a5d446bc5c
Oct 30 03:08:31.003999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:31.003966    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 03:08:31.004334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:31.004040    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 03:08:31.014145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:31.014109    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg" event={"ID":"18e0df2d-3fd3-414c-8fc0-b21092f06b1f","Type":"ContainerStarted","Data":"f464381ec540929e512c60035e1e1496bf5c7af75499dd3380e3d0a5d446bc5c"}
Oct 30 03:08:32.003850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:32.003746    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 03:08:32.003850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:32.003816    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 03:08:32.023904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:32.023863    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg" event={"ID":"18e0df2d-3fd3-414c-8fc0-b21092f06b1f","Type":"ContainerStarted","Data":"a8f621eb6c2e45b5b34c03a7a4d9bbaf1e39ab2f320fb491af4be69e22179f3e"}
Oct 30 03:08:32.066652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:32.066578    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg" podStartSLOduration=2.066556523 podStartE2EDuration="2.066556523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:32.065382903 +0000 UTC m=+1649.989591092" watchObservedRunningTime="2026-10-30 03:08:32.066556523 +0000 UTC m=+1649.990764714"
Oct 30 03:08:33.055541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:33.054609    4278 generic.go:356] "Generic (PLEG): container finished" podID="18e0df2d-3fd3-414c-8fc0-b21092f06b1f" containerID="a8f621eb6c2e45b5b34c03a7a4d9bbaf1e39ab2f320fb491af4be69e22179f3e" exitCode=0
Oct 30 03:08:33.055541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:33.054667    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg" event={"ID":"18e0df2d-3fd3-414c-8fc0-b21092f06b1f","Type":"ContainerDied","Data":"a8f621eb6c2e45b5b34c03a7a4d9bbaf1e39ab2f320fb491af4be69e22179f3e"}
Oct 30 03:08:33.218104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:33.218070    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:33.218593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:33.218569    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:33.219495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:33.219475    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:08:34.238837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:34.238807    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:34.300552 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:34.300510    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-xks-ca\") pod \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") "
Oct 30 03:08:34.300769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:34.300578    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-kube-api-access-hbj2q\" (UniqueName: \"kubernetes.io/projected/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-kube-api-access-hbj2q\") pod \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") "
Oct 30 03:08:34.301048 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:34.301021    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-openshift-ca\") pod \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\" (UID: \"18e0df2d-3fd3-414c-8fc0-b21092f06b1f\") "
Oct 30 03:08:34.301494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:34.301468    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-openshift-ca" pod "18e0df2d-3fd3-414c-8fc0-b21092f06b1f" (UID: "18e0df2d-3fd3-414c-8fc0-b21092f06b1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:34.301863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:34.301843    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:34.303003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:34.302974    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-xks-ca" pod "18e0df2d-3fd3-414c-8fc0-b21092f06b1f" (UID: "18e0df2d-3fd3-414c-8fc0-b21092f06b1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:08:34.303376 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:34.303345    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-kube-api-access-hbj2q" pod "18e0df2d-3fd3-414c-8fc0-b21092f06b1f" (UID: "18e0df2d-3fd3-414c-8fc0-b21092f06b1f"). InnerVolumeSpecName "kube-api-access-hbj2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:34.403859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:34.403819    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:34.403859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:34.403853    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbj2q\" (UniqueName: \"kubernetes.io/projected/18e0df2d-3fd3-414c-8fc0-b21092f06b1f-kube-api-access-hbj2q\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:35.071134 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:35.071100    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"
Oct 30 03:08:35.071334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:35.071137    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg" event={"ID":"18e0df2d-3fd3-414c-8fc0-b21092f06b1f","Type":"ContainerDied","Data":"f464381ec540929e512c60035e1e1496bf5c7af75499dd3380e3d0a5d446bc5c"}
Oct 30 03:08:35.071334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:35.071198    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f464381ec540929e512c60035e1e1496bf5c7af75499dd3380e3d0a5d446bc5c"
Oct 30 03:08:38.078727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:38.078687    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:08:39.716499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.716457    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"]
Oct 30 03:08:39.717213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.717198    4278 cpu_manager.go:414] "containerMap: removing container" podUID="18e0df2d-3fd3-414c-8fc0-b21092f06b1f" containerName="revoke-keys"
Oct 30 03:08:39.717265 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.717215    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e0df2d-3fd3-414c-8fc0-b21092f06b1f" containerName="revoke-keys"
Oct 30 03:08:39.717376 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.717366    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e0df2d-3fd3-414c-8fc0-b21092f06b1f" containerName="revoke-keys"
Oct 30 03:08:39.721745 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.721714    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.750036 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.749994    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"]
Oct 30 03:08:39.812275 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.812236    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7zk\" (UniqueName: \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-kube-api-access-zt7zk\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.812494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.812408    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.812494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.812478    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.812660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.812506    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.812660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.812612    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.812660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.812636    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.812833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.812714    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.812833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.812742    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.812932 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.812825    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.917809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.917763    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.918038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.917860    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.918038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.917894    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.918038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.917993    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.918038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.918017    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.918339 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.918075    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.918339 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.918112    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.918339 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.918154    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.918595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.918402    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7zk\" (UniqueName: \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-kube-api-access-zt7zk\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.925224 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.925185    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.954533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.928158    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.954533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.928176    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.954533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.939918    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.954533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.940373    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.974546 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.974382    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.978094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.978056    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.983712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.983674    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:39.984128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:39.984096    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7zk\" (UniqueName: \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-kube-api-access-zt7zk\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:40.041702 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:40.041657    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:40.366274 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:40.366234    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"]
Oct 30 03:08:40.366590 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:08:40.366563    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cc43f3_75be_43f6_8bec_59fc66f2f51f.slice/crio-ddd2f48cca042bc1cae6b29efd21d517bc9448cfb9caacd754896d1eca650247 WatchSource:0}: Error finding container ddd2f48cca042bc1cae6b29efd21d517bc9448cfb9caacd754896d1eca650247: Status 404 returned error can't find the container with id ddd2f48cca042bc1cae6b29efd21d517bc9448cfb9caacd754896d1eca650247
Oct 30 03:08:40.369809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:40.369773    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:40.369904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:40.369848    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:40.369904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:40.369876    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:41.134213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:41.134162    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" event={"ID":"d8cc43f3-75be-43f6-8bec-59fc66f2f51f","Type":"ContainerStarted","Data":"1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed"}
Oct 30 03:08:41.134796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:41.134245    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" event={"ID":"d8cc43f3-75be-43f6-8bec-59fc66f2f51f","Type":"ContainerStarted","Data":"ddd2f48cca042bc1cae6b29efd21d517bc9448cfb9caacd754896d1eca650247"}
Oct 30 03:08:41.226078 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:41.226021    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" podStartSLOduration=2.226003217 podStartE2EDuration="2.226003217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:41.202017166 +0000 UTC m=+1659.126225355" watchObservedRunningTime="2026-10-30 03:08:41.226003217 +0000 UTC m=+1659.150211405"
Oct 30 03:08:42.048592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:42.048552    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:42.049039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:42.049009    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:08:42.049263 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:42.049232    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 03:08:42.223958 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:42.223906    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"]
Oct 30 03:08:42.233257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:42.233215    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-44a8fd-hmq29"]
Oct 30 03:08:43.044039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:43.044004    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:08:43.044358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:43.044321    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 03:08:44.042515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:44.042481    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:08:44.042926 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:44.042534    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 03:08:44.203235 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:44.203200    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c6f3d6-e1cd-47a5-b2d6-af21e904408a" path="/var/lib/kubelet/pods/e0c6f3d6-e1cd-47a5-b2d6-af21e904408a/volumes"
Oct 30 03:08:45.042621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:45.042534    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:08:45.043013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:45.042612    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 03:08:46.042390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:46.042344    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:08:46.042748 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:46.042410    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 03:08:47.042963 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:47.042904    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:08:47.043532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:47.042982    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 03:08:48.047753 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:48.047717    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:48.048180 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:48.048131    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:48.049126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:48.049097    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:08:50.667721 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:50.667682    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"]
Oct 30 03:08:50.668185 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:50.667989    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy" containerID="cri-o://18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7" gracePeriod=30
Oct 30 03:08:51.340701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.340652    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"]
Oct 30 03:08:51.347129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.347100    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.367023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.366994    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"]
Oct 30 03:08:51.400066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.400018    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddzr6\" (UniqueName: \"kubernetes.io/projected/67494687-1bad-4e8a-96a0-649422fe2b94-kube-api-access-ddzr6\") pod \"maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.400318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.400295    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.400458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.400437    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.408326 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.408286    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:08:51.408513 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.408365    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:08:51.502506 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.502468    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.502742 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.502604    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.502836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.502819    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddzr6\" (UniqueName: \"kubernetes.io/projected/67494687-1bad-4e8a-96a0-649422fe2b94-kube-api-access-ddzr6\") pod \"maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.504065 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.504033    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.505636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.505610    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.524580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.524552    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddzr6\" (UniqueName: \"kubernetes.io/projected/67494687-1bad-4e8a-96a0-649422fe2b94-kube-api-access-ddzr6\") pod \"maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.664092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.663989    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:51.978016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:51.977983    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"]
Oct 30 03:08:51.979353 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:08:51.979327    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67494687_1bad_4e8a_96a0_649422fe2b94.slice/crio-2f126e06c2d4f65f58dcc7ade777a8d86fd4d9b1afd0da2af04ca314470c3828 WatchSource:0}: Error finding container 2f126e06c2d4f65f58dcc7ade777a8d86fd4d9b1afd0da2af04ca314470c3828: Status 404 returned error can't find the container with id 2f126e06c2d4f65f58dcc7ade777a8d86fd4d9b1afd0da2af04ca314470c3828
Oct 30 03:08:52.428827 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:52.428787    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k" event={"ID":"67494687-1bad-4e8a-96a0-649422fe2b94","Type":"ContainerStarted","Data":"478472dca9a9ad1c5eb6b3267d50ed4e3d1ac7af3e989e2e8c400de2044254e3"}
Oct 30 03:08:52.428997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:52.428854    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k" event={"ID":"67494687-1bad-4e8a-96a0-649422fe2b94","Type":"ContainerStarted","Data":"2f126e06c2d4f65f58dcc7ade777a8d86fd4d9b1afd0da2af04ca314470c3828"}
Oct 30 03:08:52.492749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:52.492489    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k" podStartSLOduration=1.492408243 podStartE2EDuration="1.492408243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:52.492118474 +0000 UTC m=+1670.416326664" watchObservedRunningTime="2026-10-30 03:08:52.492408243 +0000 UTC m=+1670.416616433"
Oct 30 03:08:53.439466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:53.439321    4278 generic.go:356] "Generic (PLEG): container finished" podID="67494687-1bad-4e8a-96a0-649422fe2b94" containerID="478472dca9a9ad1c5eb6b3267d50ed4e3d1ac7af3e989e2e8c400de2044254e3" exitCode=0
Oct 30 03:08:53.439466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:53.439374    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k" event={"ID":"67494687-1bad-4e8a-96a0-649422fe2b94","Type":"ContainerDied","Data":"478472dca9a9ad1c5eb6b3267d50ed4e3d1ac7af3e989e2e8c400de2044254e3"}
Oct 30 03:08:54.192540 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:08:54.192498    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:08:54.656597 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:54.656567    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:54.829057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:54.829011    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67494687-1bad-4e8a-96a0-649422fe2b94-kube-api-access-ddzr6\" (UniqueName: \"kubernetes.io/projected/67494687-1bad-4e8a-96a0-649422fe2b94-kube-api-access-ddzr6\") pod \"67494687-1bad-4e8a-96a0-649422fe2b94\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") "
Oct 30 03:08:54.829509 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:54.829478    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-openshift-ca\") pod \"67494687-1bad-4e8a-96a0-649422fe2b94\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") "
Oct 30 03:08:54.829677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:54.829534    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-xks-ca\") pod \"67494687-1bad-4e8a-96a0-649422fe2b94\" (UID: \"67494687-1bad-4e8a-96a0-649422fe2b94\") "
Oct 30 03:08:54.829916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:54.829889    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-openshift-ca" pod "67494687-1bad-4e8a-96a0-649422fe2b94" (UID: "67494687-1bad-4e8a-96a0-649422fe2b94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:54.830229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:54.830210    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:54.831356 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:54.831333    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67494687-1bad-4e8a-96a0-649422fe2b94-kube-api-access-ddzr6" pod "67494687-1bad-4e8a-96a0-649422fe2b94" (UID: "67494687-1bad-4e8a-96a0-649422fe2b94"). InnerVolumeSpecName "kube-api-access-ddzr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:54.831621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:54.831594    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-xks-ca" pod "67494687-1bad-4e8a-96a0-649422fe2b94" (UID: "67494687-1bad-4e8a-96a0-649422fe2b94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:08:54.931718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:54.931680    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67494687-1bad-4e8a-96a0-649422fe2b94-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:54.931718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:54.931709    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddzr6\" (UniqueName: \"kubernetes.io/projected/67494687-1bad-4e8a-96a0-649422fe2b94-kube-api-access-ddzr6\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:55.519080 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:55.519031    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k" event={"ID":"67494687-1bad-4e8a-96a0-649422fe2b94","Type":"ContainerDied","Data":"2f126e06c2d4f65f58dcc7ade777a8d86fd4d9b1afd0da2af04ca314470c3828"}
Oct 30 03:08:55.519321 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:55.519112    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f126e06c2d4f65f58dcc7ade777a8d86fd4d9b1afd0da2af04ca314470c3828"
Oct 30 03:08:55.519321 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:55.519187    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"
Oct 30 03:08:56.203859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.203822    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:08:56.340161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340055    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-kube-api-access-j8k6s\" (UniqueName: \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-kube-api-access-j8k6s\") pod \"0e5c2e8d-d618-450e-81ff-04c418279c52\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") "
Oct 30 03:08:56.340161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340115    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-socket\") pod \"0e5c2e8d-d618-450e-81ff-04c418279c52\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") "
Oct 30 03:08:56.340409 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340306    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e5c2e8d-d618-450e-81ff-04c418279c52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5c2e8d-d618-450e-81ff-04c418279c52-istiod-ca-cert\") pod \"0e5c2e8d-d618-450e-81ff-04c418279c52\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") "
Oct 30 03:08:56.340409 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340343    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-envoy\") pod \"0e5c2e8d-d618-450e-81ff-04c418279c52\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") "
Oct 30 03:08:56.340532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340411    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-token\" (UniqueName: \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-token\") pod \"0e5c2e8d-d618-450e-81ff-04c418279c52\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") "
Oct 30 03:08:56.340589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340526    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-socket" pod "0e5c2e8d-d618-450e-81ff-04c418279c52" (UID: "0e5c2e8d-d618-450e-81ff-04c418279c52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:56.340589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340577    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-credential-socket\") pod \"0e5c2e8d-d618-450e-81ff-04c418279c52\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") "
Oct 30 03:08:56.340718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340698    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-podinfo\") pod \"0e5c2e8d-d618-450e-81ff-04c418279c52\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") "
Oct 30 03:08:56.340778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340714    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5c2e8d-d618-450e-81ff-04c418279c52-istiod-ca-cert" pod "0e5c2e8d-d618-450e-81ff-04c418279c52" (UID: "0e5c2e8d-d618-450e-81ff-04c418279c52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:56.340778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340761    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-data\") pod \"0e5c2e8d-d618-450e-81ff-04c418279c52\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") "
Oct 30 03:08:56.340963 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.340943    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-certs\") pod \"0e5c2e8d-d618-450e-81ff-04c418279c52\" (UID: \"0e5c2e8d-d618-450e-81ff-04c418279c52\") "
Oct 30 03:08:56.341080 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.341058    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-credential-socket" pod "0e5c2e8d-d618-450e-81ff-04c418279c52" (UID: "0e5c2e8d-d618-450e-81ff-04c418279c52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:56.341679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.341647    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-data" pod "0e5c2e8d-d618-450e-81ff-04c418279c52" (UID: "0e5c2e8d-d618-450e-81ff-04c418279c52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:56.341898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.341874    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-certs" pod "0e5c2e8d-d618-450e-81ff-04c418279c52" (UID: "0e5c2e8d-d618-450e-81ff-04c418279c52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:56.341982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.341916    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.341982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.341940    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.341982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.341956    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.341982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.341971    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5c2e8d-d618-450e-81ff-04c418279c52-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.343030 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.342980    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-kube-api-access-j8k6s" pod "0e5c2e8d-d618-450e-81ff-04c418279c52" (UID: "0e5c2e8d-d618-450e-81ff-04c418279c52"). InnerVolumeSpecName "kube-api-access-j8k6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:56.343161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.343109    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-envoy" pod "0e5c2e8d-d618-450e-81ff-04c418279c52" (UID: "0e5c2e8d-d618-450e-81ff-04c418279c52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:56.343788 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.343756    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-podinfo" pod "0e5c2e8d-d618-450e-81ff-04c418279c52" (UID: "0e5c2e8d-d618-450e-81ff-04c418279c52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:08:56.343920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.343832    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-token" pod "0e5c2e8d-d618-450e-81ff-04c418279c52" (UID: "0e5c2e8d-d618-450e-81ff-04c418279c52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:56.443539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.443494    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.443539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.443533    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.443539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.443543    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5c2e8d-d618-450e-81ff-04c418279c52-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.443789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.443551    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5c2e8d-d618-450e-81ff-04c418279c52-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.443789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.443562    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8k6s\" (UniqueName: \"kubernetes.io/projected/0e5c2e8d-d618-450e-81ff-04c418279c52-kube-api-access-j8k6s\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.529108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.529065    4278 generic.go:356] "Generic (PLEG): container finished" podID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerID="18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7" exitCode=0
Oct 30 03:08:56.529322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.529114    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"
Oct 30 03:08:56.529322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.529146    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" event={"ID":"0e5c2e8d-d618-450e-81ff-04c418279c52","Type":"ContainerDied","Data":"18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7"}
Oct 30 03:08:56.529322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.529216    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b" event={"ID":"0e5c2e8d-d618-450e-81ff-04c418279c52","Type":"ContainerDied","Data":"ae5025f014973756a25046836b0d679d890768263f5b33b0dc5702e9d45106a9"}
Oct 30 03:08:56.529322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.529247    4278 scope.go:122] "RemoveContainer" containerID="18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7"
Oct 30 03:08:56.544500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.543356    4278 scope.go:122] "RemoveContainer" containerID="97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a"
Oct 30 03:08:56.559205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.559177    4278 scope.go:122] "RemoveContainer" containerID="18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7"
Oct 30 03:08:56.559616 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:08:56.559591    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7\": container with ID starting with 18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7 not found: ID does not exist" containerID="18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7"
Oct 30 03:08:56.559721 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.559627    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7"} err="failed to get container status \"18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7\": rpc error: code = NotFound desc = could not find container \"18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7\": container with ID starting with 18d2d554b488ed386836ebe6426af34ee6102c894f3b3e552709403e5dd8f9a7 not found: ID does not exist"
Oct 30 03:08:56.559721 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.559649    4278 scope.go:122] "RemoveContainer" containerID="97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a"
Oct 30 03:08:56.560062 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:08:56.560039    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a\": container with ID starting with 97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a not found: ID does not exist" containerID="97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a"
Oct 30 03:08:56.560135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.560069    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a"} err="failed to get container status \"97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a\": rpc error: code = NotFound desc = could not find container \"97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a\": container with ID starting with 97567b7823ce17601b31ad33a3539aeb09aa4b2160f79c674e492c0b3854677a not found: ID does not exist"
Oct 30 03:08:56.586276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.586176    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"]
Oct 30 03:08:56.608057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.606063    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be9f2a59-qgjl2"]
Oct 30 03:08:56.614565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.614525    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"]
Oct 30 03:08:56.618401 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:56.618371    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-44a8fd-openshift-default-5bb9d64b95-8mv2b"]
Oct 30 03:08:58.202276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:58.202240    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" path="/var/lib/kubelet/pods/0e5c2e8d-d618-450e-81ff-04c418279c52/volumes"
Oct 30 03:08:58.202925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:08:58.202907    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a527b68-095f-4b83-abbc-df94b88dff35" path="/var/lib/kubelet/pods/8a527b68-095f-4b83-abbc-df94b88dff35/volumes"
Oct 30 03:09:02.129217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:02.129165    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"]
Oct 30 03:09:06.046126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:06.046083    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"]
Oct 30 03:09:06.055095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:06.055057    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1abc2-vx8fk"]
Oct 30 03:09:06.207257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:06.207218    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a61bfe15-f42d-424a-9376-37ec7d48581d" path="/var/lib/kubelet/pods/a61bfe15-f42d-424a-9376-37ec7d48581d/volumes"
Oct 30 03:09:17.160248 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.160174    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5" podUID="cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" containerName="main" containerID="cri-o://9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4" gracePeriod=45
Oct 30 03:09:17.334348 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.334300    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5" podUID="cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.117:8000/ready\": dial tcp 10.134.0.117:8000: connect: connection refused"
Oct 30 03:09:17.516653 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.516624    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:09:17.656448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.655673    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-model-cache\") pod \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") "
Oct 30 03:09:17.656448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.655756    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-kube-api-access-kqb7s\" (UniqueName: \"kubernetes.io/projected/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-kube-api-access-kqb7s\") pod \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") "
Oct 30 03:09:17.656448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.655882    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-dshm\") pod \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") "
Oct 30 03:09:17.656448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.655931    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tmp-dir\") pod \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") "
Oct 30 03:09:17.656448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.656038    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tls-certs\") pod \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") "
Oct 30 03:09:17.656448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.656144    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-home\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-home\") pod \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\" (UID: \"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e\") "
Oct 30 03:09:17.656448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.656364    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tmp-dir" pod "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" (UID: "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:17.656448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.656378    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-model-cache" pod "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" (UID: "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:17.656999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.656672    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-home" pod "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" (UID: "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:17.656999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.656926    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:17.656999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.656951    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:17.656999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.656969    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:17.666449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.660945    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tls-certs" pod "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" (UID: "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:09:17.666449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.661115    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-kube-api-access-kqb7s" pod "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" (UID: "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e"). InnerVolumeSpecName "kube-api-access-kqb7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:17.674636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.674600    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-dshm" pod "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" (UID: "cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:17.777448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.764576    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:17.777448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.764623    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:17.777448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:17.764743    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqb7s\" (UniqueName: \"kubernetes.io/projected/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e-kube-api-access-kqb7s\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:18.099530 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:18.071472    4278 generic.go:356] "Generic (PLEG): container finished" podID="cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" containerID="9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4" exitCode=0
Oct 30 03:09:18.099530 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:18.071786    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5" event={"ID":"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e","Type":"ContainerDied","Data":"9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4"}
Oct 30 03:09:18.099530 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:18.071832    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5" event={"ID":"cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e","Type":"ContainerDied","Data":"8edb1310059b82a70ccb68ee8c0eec86b3409a95e22d1ac9af90fde3985ca6c8"}
Oct 30 03:09:18.099530 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:18.071856    4278 scope.go:122] "RemoveContainer" containerID="9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4"
Oct 30 03:09:18.099530 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:18.072065    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"
Oct 30 03:09:18.260720 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:18.260584    4278 scope.go:122] "RemoveContainer" containerID="9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4"
Oct 30 03:09:18.261769 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:09:18.261684    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4\": container with ID starting with 9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4 not found: ID does not exist" containerID="9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4"
Oct 30 03:09:18.261769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:18.261733    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4"} err="failed to get container status \"9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4\": rpc error: code = NotFound desc = could not find container \"9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4\": container with ID starting with 9ea230523462545c8242b986f28268e93887c612886ac13c9fef6613d53595e4 not found: ID does not exist"
Oct 30 03:09:18.295692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:18.295642    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"]
Oct 30 03:09:18.312743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:18.312703    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-be9f2a59/e2e-lifecycle-model-be9f2a59-kserve-577d88d697-f47x5"]
Oct 30 03:09:20.202591 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:20.202558    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" path="/var/lib/kubelet/pods/cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e/volumes"
Oct 30 03:09:24.345755 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:09:24.345707    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:09:37.358455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:37.354563    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:09:45.427351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:45.427314    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"]
Oct 30 03:09:45.439160 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:45.439127    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3337a-mcwlg"]
Oct 30 03:09:46.145172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:46.145134    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"]
Oct 30 03:09:46.145458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:46.145387    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy" containerID="cri-o://e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394" gracePeriod=30
Oct 30 03:09:46.171378 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:46.171330    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"]
Oct 30 03:09:46.171810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:46.171687    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy" containerID="cri-o://1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed" gracePeriod=30
Oct 30 03:09:46.178114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:46.178085    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"]
Oct 30 03:09:46.178360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:46.178335    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy" containerID="cri-o://c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093" gracePeriod=30
Oct 30 03:09:46.201394 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:46.201360    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e0df2d-3fd3-414c-8fc0-b21092f06b1f" path="/var/lib/kubelet/pods/18e0df2d-3fd3-414c-8fc0-b21092f06b1f/volumes"
Oct 30 03:09:46.852262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:46.852224    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"]
Oct 30 03:09:46.852739 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:46.852462    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerName="istio-proxy" containerID="cri-o://4b48efecd1f2208f61895e4d5d67fdc8611abbc76b8bc6416231bf994a84a01b" gracePeriod=30
Oct 30 03:09:47.056239 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.056197    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"]
Oct 30 03:09:47.056550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.056515    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy" containerID="cri-o://7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f" gracePeriod=30
Oct 30 03:09:47.571831 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.571792    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"]
Oct 30 03:09:47.572108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.572057    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy" containerID="cri-o://0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181" gracePeriod=30
Oct 30 03:09:47.655706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.655666    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"]
Oct 30 03:09:47.656499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656440    4278 cpu_manager.go:414] "containerMap: removing container" podUID="67494687-1bad-4e8a-96a0-649422fe2b94" containerName="revoke-keys"
Oct 30 03:09:47.656499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656464    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67494687-1bad-4e8a-96a0-649422fe2b94" containerName="revoke-keys"
Oct 30 03:09:47.656598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656535    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy"
Oct 30 03:09:47.656598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656541    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy"
Oct 30 03:09:47.656598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656550    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy"
Oct 30 03:09:47.656598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656555    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy"
Oct 30 03:09:47.656598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656570    4278 cpu_manager.go:414] "containerMap: removing container" podUID="cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" containerName="main"
Oct 30 03:09:47.656598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656575    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" containerName="main"
Oct 30 03:09:47.656775 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656715    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy"
Oct 30 03:09:47.656775 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656733    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5c2e8d-d618-450e-81ff-04c418279c52" containerName="istio-proxy"
Oct 30 03:09:47.656775 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656741    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb5e0d4-a7b5-4ce4-b811-ae9219358a0e" containerName="main"
Oct 30 03:09:47.656775 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.656747    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="67494687-1bad-4e8a-96a0-649422fe2b94" containerName="revoke-keys"
Oct 30 03:09:47.661786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.661763    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:47.686985 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.686930    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"]
Oct 30 03:09:47.687262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.687230    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:09:47.687595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.687395    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:09:47.821098 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.821052    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frmvx\" (UniqueName: \"kubernetes.io/projected/9f884cf6-bf41-45f7-837b-63b591cd2af2-kube-api-access-frmvx\") pod \"maas-api-revoke-sub-c6058e71-pxpz5\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:47.821318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.821230    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6058e71-pxpz5\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:47.821593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.821564    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6058e71-pxpz5\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:47.915690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.912342    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:47.915690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.912565    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:47.923663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.923629    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6058e71-pxpz5\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:47.923835 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.923814    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frmvx\" (UniqueName: \"kubernetes.io/projected/9f884cf6-bf41-45f7-837b-63b591cd2af2-kube-api-access-frmvx\") pod \"maas-api-revoke-sub-c6058e71-pxpz5\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:47.923881 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.923854    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6058e71-pxpz5\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:47.924634 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.924609    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6058e71-pxpz5\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:47.926226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.926198    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6058e71-pxpz5\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:47.944215 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:47.944184    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmvx\" (UniqueName: \"kubernetes.io/projected/9f884cf6-bf41-45f7-837b-63b591cd2af2-kube-api-access-frmvx\") pod \"maas-api-revoke-sub-c6058e71-pxpz5\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:48.000921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:48.000885    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:48.051533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:48.051385    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:48.051533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:48.051473    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:48.220579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:48.220494    4278 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:48.220579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:48.220567    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:48.311956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:48.311919    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"]
Oct 30 03:09:48.313042 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:09:48.313007    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f884cf6_bf41_45f7_837b_63b591cd2af2.slice/crio-8409e2e665407bbb6f5482e5e60b30d09279deb9350ed9127b06ae46df84b236 WatchSource:0}: Error finding container 8409e2e665407bbb6f5482e5e60b30d09279deb9350ed9127b06ae46df84b236: Status 404 returned error can't find the container with id 8409e2e665407bbb6f5482e5e60b30d09279deb9350ed9127b06ae46df84b236
Oct 30 03:09:48.578965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:48.578928    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5" event={"ID":"9f884cf6-bf41-45f7-837b-63b591cd2af2","Type":"ContainerStarted","Data":"8409e2e665407bbb6f5482e5e60b30d09279deb9350ed9127b06ae46df84b236"}
Oct 30 03:09:49.588968 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:49.588925    4278 generic.go:356] "Generic (PLEG): container finished" podID="9f884cf6-bf41-45f7-837b-63b591cd2af2" containerID="49caa015ba276cf880e5a65b1b7baa9b5e5f9fad0b019f8d0b32170a83092b3c" exitCode=0
Oct 30 03:09:49.589412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:49.589013    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5" event={"ID":"9f884cf6-bf41-45f7-837b-63b591cd2af2","Type":"ContainerDied","Data":"49caa015ba276cf880e5a65b1b7baa9b5e5f9fad0b019f8d0b32170a83092b3c"}
Oct 30 03:09:50.776216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:50.776189    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:50.879643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:50.879537    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f884cf6-bf41-45f7-837b-63b591cd2af2-kube-api-access-frmvx\" (UniqueName: \"kubernetes.io/projected/9f884cf6-bf41-45f7-837b-63b591cd2af2-kube-api-access-frmvx\") pod \"9f884cf6-bf41-45f7-837b-63b591cd2af2\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") "
Oct 30 03:09:50.880992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:50.880947    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-openshift-ca" pod "9f884cf6-bf41-45f7-837b-63b591cd2af2" (UID: "9f884cf6-bf41-45f7-837b-63b591cd2af2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:50.898158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:50.898107    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f884cf6-bf41-45f7-837b-63b591cd2af2-kube-api-access-frmvx" pod "9f884cf6-bf41-45f7-837b-63b591cd2af2" (UID: "9f884cf6-bf41-45f7-837b-63b591cd2af2"). InnerVolumeSpecName "kube-api-access-frmvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:50.899753 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:50.881040    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-openshift-ca\") pod \"9f884cf6-bf41-45f7-837b-63b591cd2af2\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") "
Oct 30 03:09:50.900057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:50.900031    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-xks-ca\") pod \"9f884cf6-bf41-45f7-837b-63b591cd2af2\" (UID: \"9f884cf6-bf41-45f7-837b-63b591cd2af2\") "
Oct 30 03:09:50.901121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:50.901091    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:50.901278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:50.901255    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frmvx\" (UniqueName: \"kubernetes.io/projected/9f884cf6-bf41-45f7-837b-63b591cd2af2-kube-api-access-frmvx\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:50.903829 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:50.903797    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-xks-ca" pod "9f884cf6-bf41-45f7-837b-63b591cd2af2" (UID: "9f884cf6-bf41-45f7-837b-63b591cd2af2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:09:51.003126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.003088    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f884cf6-bf41-45f7-837b-63b591cd2af2-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.340521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.340058    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:51.340521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.340123    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:51.467694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.467660    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:09:51.512309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.512264    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-certs\") pod \"c7588026-2174-4904-a2af-8e4cdf2873f5\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") "
Oct 30 03:09:51.512488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.512356    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-envoy\") pod \"c7588026-2174-4904-a2af-8e4cdf2873f5\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") "
Oct 30 03:09:51.512488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.512407    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7588026-2174-4904-a2af-8e4cdf2873f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7588026-2174-4904-a2af-8e4cdf2873f5-istiod-ca-cert\") pod \"c7588026-2174-4904-a2af-8e4cdf2873f5\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") "
Oct 30 03:09:51.512630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.512549    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-data\") pod \"c7588026-2174-4904-a2af-8e4cdf2873f5\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") "
Oct 30 03:09:51.512630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.512617    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-credential-socket\") pod \"c7588026-2174-4904-a2af-8e4cdf2873f5\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") "
Oct 30 03:09:51.512749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.512708    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-kube-api-access-qq26d\" (UniqueName: \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-kube-api-access-qq26d\") pod \"c7588026-2174-4904-a2af-8e4cdf2873f5\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") "
Oct 30 03:09:51.512807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.512767    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-podinfo\") pod \"c7588026-2174-4904-a2af-8e4cdf2873f5\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") "
Oct 30 03:09:51.512860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.512813    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-socket\") pod \"c7588026-2174-4904-a2af-8e4cdf2873f5\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") "
Oct 30 03:09:51.512860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.512831    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-token\" (UniqueName: \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-token\") pod \"c7588026-2174-4904-a2af-8e4cdf2873f5\" (UID: \"c7588026-2174-4904-a2af-8e4cdf2873f5\") "
Oct 30 03:09:51.513257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.512607    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-certs" pod "c7588026-2174-4904-a2af-8e4cdf2873f5" (UID: "c7588026-2174-4904-a2af-8e4cdf2873f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.513257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.513000    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7588026-2174-4904-a2af-8e4cdf2873f5-istiod-ca-cert" pod "c7588026-2174-4904-a2af-8e4cdf2873f5" (UID: "c7588026-2174-4904-a2af-8e4cdf2873f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:51.513527 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.513507    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.513610 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.513532    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7588026-2174-4904-a2af-8e4cdf2873f5-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.514180 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.513748    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-socket" pod "c7588026-2174-4904-a2af-8e4cdf2873f5" (UID: "c7588026-2174-4904-a2af-8e4cdf2873f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.514180 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.513899    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-credential-socket" pod "c7588026-2174-4904-a2af-8e4cdf2873f5" (UID: "c7588026-2174-4904-a2af-8e4cdf2873f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.516770 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.516703    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-data" pod "c7588026-2174-4904-a2af-8e4cdf2873f5" (UID: "c7588026-2174-4904-a2af-8e4cdf2873f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.519068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.518967    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-podinfo" pod "c7588026-2174-4904-a2af-8e4cdf2873f5" (UID: "c7588026-2174-4904-a2af-8e4cdf2873f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:51.519068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.518996    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-token" pod "c7588026-2174-4904-a2af-8e4cdf2873f5" (UID: "c7588026-2174-4904-a2af-8e4cdf2873f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.519068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.519016    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-envoy" pod "c7588026-2174-4904-a2af-8e4cdf2873f5" (UID: "c7588026-2174-4904-a2af-8e4cdf2873f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.519068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.519039    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-kube-api-access-qq26d" pod "c7588026-2174-4904-a2af-8e4cdf2873f5" (UID: "c7588026-2174-4904-a2af-8e4cdf2873f5"). InnerVolumeSpecName "kube-api-access-qq26d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.525866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.525841    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:09:51.545498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.545348    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:09:51.608369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.608335    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"
Oct 30 03:09:51.609228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.609193    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5" event={"ID":"9f884cf6-bf41-45f7-837b-63b591cd2af2","Type":"ContainerDied","Data":"8409e2e665407bbb6f5482e5e60b30d09279deb9350ed9127b06ae46df84b236"}
Oct 30 03:09:51.609481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.609461    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8409e2e665407bbb6f5482e5e60b30d09279deb9350ed9127b06ae46df84b236"
Oct 30 03:09:51.613519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.613467    4278 generic.go:356] "Generic (PLEG): container finished" podID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerID="1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed" exitCode=0
Oct 30 03:09:51.613651 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.613529    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"
Oct 30 03:09:51.613651 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.613559    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" event={"ID":"d8cc43f3-75be-43f6-8bec-59fc66f2f51f","Type":"ContainerDied","Data":"1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed"}
Oct 30 03:09:51.614021 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614002    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-kube-api-access-zt7zk\" (UniqueName: \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-kube-api-access-zt7zk\") pod \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") "
Oct 30 03:09:51.614105 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614051    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-token\" (UniqueName: \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-token\") pod \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") "
Oct 30 03:09:51.614105 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614090    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-kube-api-access-brfgc\" (UniqueName: \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-kube-api-access-brfgc\") pod \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") "
Oct 30 03:09:51.614220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614122    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-credential-socket\") pod \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") "
Oct 30 03:09:51.614220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614151    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istiod-ca-cert\") pod \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") "
Oct 30 03:09:51.614220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614201    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-envoy\") pod \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") "
Oct 30 03:09:51.614383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614229    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-certs\") pod \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") "
Oct 30 03:09:51.614383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614253    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-data\") pod \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") "
Oct 30 03:09:51.614383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614277    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-certs\") pod \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") "
Oct 30 03:09:51.614383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614326    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-envoy\") pod \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") "
Oct 30 03:09:51.614622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614396    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-socket\") pod \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") "
Oct 30 03:09:51.614622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614465    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-podinfo\") pod \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") "
Oct 30 03:09:51.614622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614493    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-podinfo\") pod \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") "
Oct 30 03:09:51.614622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614566    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-data\") pod \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") "
Oct 30 03:09:51.614622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614620    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-token\" (UniqueName: \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-token\") pod \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") "
Oct 30 03:09:51.615000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614703    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-socket\") pod \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\" (UID: \"d8cc43f3-75be-43f6-8bec-59fc66f2f51f\") "
Oct 30 03:09:51.615000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614754    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-certs" pod "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" (UID: "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.615000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614813    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istiod-ca-cert\") pod \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") "
Oct 30 03:09:51.615000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614877    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-credential-socket\") pod \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\" (UID: \"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02\") "
Oct 30 03:09:51.615000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614952    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-certs" pod "d8cc43f3-75be-43f6-8bec-59fc66f2f51f" (UID: "d8cc43f3-75be-43f6-8bec-59fc66f2f51f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.615000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.614974    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf" event={"ID":"d8cc43f3-75be-43f6-8bec-59fc66f2f51f","Type":"ContainerDied","Data":"ddd2f48cca042bc1cae6b29efd21d517bc9448cfb9caacd754896d1eca650247"}
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615036    4278 scope.go:122] "RemoveContainer" containerID="1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed"
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615128    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-data" pod "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" (UID: "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615700    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq26d\" (UniqueName: \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-kube-api-access-qq26d\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615722    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615736    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615765    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615777    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615790    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615804    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615818    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615830    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.615898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.615844    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7588026-2174-4904-a2af-8e4cdf2873f5-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.616510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.616025    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-socket" pod "d8cc43f3-75be-43f6-8bec-59fc66f2f51f" (UID: "d8cc43f3-75be-43f6-8bec-59fc66f2f51f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.616510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.616055    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-credential-socket" pod "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" (UID: "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.616510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.616182    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istiod-ca-cert" pod "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" (UID: "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:51.616806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.616784    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-data" pod "d8cc43f3-75be-43f6-8bec-59fc66f2f51f" (UID: "d8cc43f3-75be-43f6-8bec-59fc66f2f51f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.623497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.618533    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-credential-socket" pod "d8cc43f3-75be-43f6-8bec-59fc66f2f51f" (UID: "d8cc43f3-75be-43f6-8bec-59fc66f2f51f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.623497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.619297    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-socket" pod "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" (UID: "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.623497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.620911    4278 generic.go:356] "Generic (PLEG): container finished" podID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerID="c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093" exitCode=0
Oct 30 03:09:51.623497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.621084    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" event={"ID":"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02","Type":"ContainerDied","Data":"c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093"}
Oct 30 03:09:51.623497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.621132    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4" event={"ID":"1dc6b7b8-98a4-46ca-b4d8-710f8a576a02","Type":"ContainerDied","Data":"c02931dad31bc596f4a0934be88a757d7074bf4e1fcb78002f61000ad8fe7075"}
Oct 30 03:09:51.623497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.621234    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"
Oct 30 03:09:51.623497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.623459    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istiod-ca-cert" pod "d8cc43f3-75be-43f6-8bec-59fc66f2f51f" (UID: "d8cc43f3-75be-43f6-8bec-59fc66f2f51f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:51.628544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.624183    4278 generic.go:356] "Generic (PLEG): container finished" podID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerID="e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394" exitCode=0
Oct 30 03:09:51.628544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.624298    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" event={"ID":"c7588026-2174-4904-a2af-8e4cdf2873f5","Type":"ContainerDied","Data":"e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394"}
Oct 30 03:09:51.628544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.624340    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85" event={"ID":"c7588026-2174-4904-a2af-8e4cdf2873f5","Type":"ContainerDied","Data":"4d416a12c38f1cdca9db177cf1dba2babd823d595318c28c0ee32469a9292f87"}
Oct 30 03:09:51.628544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.624417    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"
Oct 30 03:09:51.629052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.629012    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-envoy" pod "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" (UID: "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.632719 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.632690    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-kube-api-access-zt7zk" pod "d8cc43f3-75be-43f6-8bec-59fc66f2f51f" (UID: "d8cc43f3-75be-43f6-8bec-59fc66f2f51f"). InnerVolumeSpecName "kube-api-access-zt7zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.632914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.632870    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-kube-api-access-brfgc" pod "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" (UID: "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02"). InnerVolumeSpecName "kube-api-access-brfgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.632914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.632898    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-token" pod "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" (UID: "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.633149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.633099    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-envoy" pod "d8cc43f3-75be-43f6-8bec-59fc66f2f51f" (UID: "d8cc43f3-75be-43f6-8bec-59fc66f2f51f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.633149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.633109    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-podinfo" pod "d8cc43f3-75be-43f6-8bec-59fc66f2f51f" (UID: "d8cc43f3-75be-43f6-8bec-59fc66f2f51f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:51.634391 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.634362    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-podinfo" pod "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" (UID: "1dc6b7b8-98a4-46ca-b4d8-710f8a576a02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:51.636224 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.636203    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-token" pod "d8cc43f3-75be-43f6-8bec-59fc66f2f51f" (UID: "d8cc43f3-75be-43f6-8bec-59fc66f2f51f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.642069 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.642048    4278 scope.go:122] "RemoveContainer" containerID="1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed"
Oct 30 03:09:51.642446 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:09:51.642410    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed\": container with ID starting with 1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed not found: ID does not exist" containerID="1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed"
Oct 30 03:09:51.642526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.642456    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed"} err="failed to get container status \"1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed\": rpc error: code = NotFound desc = could not find container \"1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed\": container with ID starting with 1b176ee8a9c7c0b12dee875f655b919f3654ece3a19219c7170ac225b66e08ed not found: ID does not exist"
Oct 30 03:09:51.642526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.642475    4278 scope.go:122] "RemoveContainer" containerID="c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093"
Oct 30 03:09:51.655986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.655835    4278 scope.go:122] "RemoveContainer" containerID="c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093"
Oct 30 03:09:51.656373 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:09:51.656347    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093\": container with ID starting with c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093 not found: ID does not exist" containerID="c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093"
Oct 30 03:09:51.656487 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.656384    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093"} err="failed to get container status \"c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093\": rpc error: code = NotFound desc = could not find container \"c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093\": container with ID starting with c877a359d3b8fed896c0ecafa2f355a8c543f06c39d4cd8100bbe1f42d61f093 not found: ID does not exist"
Oct 30 03:09:51.656487 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.656407    4278 scope.go:122] "RemoveContainer" containerID="e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394"
Oct 30 03:09:51.676755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.676727    4278 scope.go:122] "RemoveContainer" containerID="e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394"
Oct 30 03:09:51.677177 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:09:51.677153    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394\": container with ID starting with e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394 not found: ID does not exist" containerID="e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394"
Oct 30 03:09:51.677253 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.677188    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394"} err="failed to get container status \"e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394\": rpc error: code = NotFound desc = could not find container \"e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394\": container with ID starting with e826eadc04da03a2efe90af766446d97204063ba5cf39661bf395e4152480394 not found: ID does not exist"
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718614    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718646    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt7zk\" (UniqueName: \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-kube-api-access-zt7zk\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718657    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718666    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfgc\" (UniqueName: \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-kube-api-access-brfgc\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718676    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718684    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718693    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718701    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718708    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718716    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.718717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718724    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.719419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718732    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.719419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718741    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.719419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718749    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8cc43f3-75be-43f6-8bec-59fc66f2f51f-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.719419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.718757    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.726143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.726071    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"]
Oct 30 03:09:51.731605 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.731565    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-c4c85"]
Oct 30 03:09:51.967787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.967750    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"]
Oct 30 03:09:51.972899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:51.972831    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-52nxf"]
Oct 30 03:09:52.007748 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.007708    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"]
Oct 30 03:09:52.011578 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.011548    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-752j4"]
Oct 30 03:09:52.203940 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.203906    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" path="/var/lib/kubelet/pods/1dc6b7b8-98a4-46ca-b4d8-710f8a576a02/volumes"
Oct 30 03:09:52.204534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.204519    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" path="/var/lib/kubelet/pods/c7588026-2174-4904-a2af-8e4cdf2873f5/volumes"
Oct 30 03:09:52.205016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.205003    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" path="/var/lib/kubelet/pods/d8cc43f3-75be-43f6-8bec-59fc66f2f51f/volumes"
Oct 30 03:09:52.369034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.369002    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 03:09:52.467386 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.467347    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-data\") pod \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") "
Oct 30 03:09:52.467633 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.467459    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-podinfo\") pod \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") "
Oct 30 03:09:52.467633 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.467548    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-certs\") pod \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") "
Oct 30 03:09:52.467933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.467681    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-credential-socket\") pod \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") "
Oct 30 03:09:52.467933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.467748    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-token\" (UniqueName: \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-token\") pod \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") "
Oct 30 03:09:52.467933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.467784    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-kube-api-access-lx6n6\" (UniqueName: \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-kube-api-access-lx6n6\") pod \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") "
Oct 30 03:09:52.467933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.467871    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-envoy\") pod \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") "
Oct 30 03:09:52.468205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.467956    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istiod-ca-cert\") pod \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") "
Oct 30 03:09:52.468205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.468128    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-socket\") pod \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\" (UID: \"04e529a9-14f9-4f80-96a2-2c56fd4abe64\") "
Oct 30 03:09:52.471573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.471487    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-certs" pod "04e529a9-14f9-4f80-96a2-2c56fd4abe64" (UID: "04e529a9-14f9-4f80-96a2-2c56fd4abe64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.471786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.471624    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istiod-ca-cert" pod "04e529a9-14f9-4f80-96a2-2c56fd4abe64" (UID: "04e529a9-14f9-4f80-96a2-2c56fd4abe64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:52.472049 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.471960    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-data" pod "04e529a9-14f9-4f80-96a2-2c56fd4abe64" (UID: "04e529a9-14f9-4f80-96a2-2c56fd4abe64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.472579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.472552    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-credential-socket" pod "04e529a9-14f9-4f80-96a2-2c56fd4abe64" (UID: "04e529a9-14f9-4f80-96a2-2c56fd4abe64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.473486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.473413    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-token" pod "04e529a9-14f9-4f80-96a2-2c56fd4abe64" (UID: "04e529a9-14f9-4f80-96a2-2c56fd4abe64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:52.474178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.473776    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-podinfo" pod "04e529a9-14f9-4f80-96a2-2c56fd4abe64" (UID: "04e529a9-14f9-4f80-96a2-2c56fd4abe64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:52.474748 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.474700    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-envoy" pod "04e529a9-14f9-4f80-96a2-2c56fd4abe64" (UID: "04e529a9-14f9-4f80-96a2-2c56fd4abe64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.474997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.474972    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-kube-api-access-lx6n6" pod "04e529a9-14f9-4f80-96a2-2c56fd4abe64" (UID: "04e529a9-14f9-4f80-96a2-2c56fd4abe64"). InnerVolumeSpecName "kube-api-access-lx6n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:52.475150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.475125    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-socket" pod "04e529a9-14f9-4f80-96a2-2c56fd4abe64" (UID: "04e529a9-14f9-4f80-96a2-2c56fd4abe64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.579216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.579177    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.579216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.579217    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.579571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.579233    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.579571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.579251    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.579571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.579265    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.579571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.579279    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.579571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.579293    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.579571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.579306    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx6n6\" (UniqueName: \"kubernetes.io/projected/04e529a9-14f9-4f80-96a2-2c56fd4abe64-kube-api-access-lx6n6\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.579571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.579320    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e529a9-14f9-4f80-96a2-2c56fd4abe64-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.637701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.637665    4278 generic.go:356] "Generic (PLEG): container finished" podID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerID="4b48efecd1f2208f61895e4d5d67fdc8611abbc76b8bc6416231bf994a84a01b" exitCode=0
Oct 30 03:09:52.637930 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.637729    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" event={"ID":"a82519a5-4fac-40c4-a8c2-230c028d2934","Type":"ContainerDied","Data":"4b48efecd1f2208f61895e4d5d67fdc8611abbc76b8bc6416231bf994a84a01b"}
Oct 30 03:09:52.639652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.639624    4278 generic.go:356] "Generic (PLEG): container finished" podID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerID="7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f" exitCode=0
Oct 30 03:09:52.639803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.639697    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"
Oct 30 03:09:52.639803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.639711    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" event={"ID":"04e529a9-14f9-4f80-96a2-2c56fd4abe64","Type":"ContainerDied","Data":"7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f"}
Oct 30 03:09:52.639803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.639781    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7" event={"ID":"04e529a9-14f9-4f80-96a2-2c56fd4abe64","Type":"ContainerDied","Data":"847165ddd93acb98319ee9f0275fa61aa2ac3ef04f3aabf489f02e5122e05781"}
Oct 30 03:09:52.639935 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.639818    4278 scope.go:122] "RemoveContainer" containerID="7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f"
Oct 30 03:09:52.692129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.692098    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 03:09:52.694375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.694322    4278 scope.go:122] "RemoveContainer" containerID="7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f"
Oct 30 03:09:52.694952 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:09:52.694929    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f\": container with ID starting with 7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f not found: ID does not exist" containerID="7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f"
Oct 30 03:09:52.695058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.694961    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f"} err="failed to get container status \"7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f\": rpc error: code = NotFound desc = could not find container \"7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f\": container with ID starting with 7c288c2faccbfe757184fda8e5285dc5aaa6e6a84fac8207d5f659d96d3a6d4f not found: ID does not exist"
Oct 30 03:09:52.699567 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.699538    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"]
Oct 30 03:09:52.705995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.705967    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-mrwb7"]
Oct 30 03:09:52.707596 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:09:52.707569    4278 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e529a9_14f9_4f80_96a2_2c56fd4abe64.slice/crio-847165ddd93acb98319ee9f0275fa61aa2ac3ef04f3aabf489f02e5122e05781\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e529a9_14f9_4f80_96a2_2c56fd4abe64.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 03:09:52.781281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.781243    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-envoy\") pod \"a82519a5-4fac-40c4-a8c2-230c028d2934\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") "
Oct 30 03:09:52.781497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.781307    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-credential-socket\") pod \"a82519a5-4fac-40c4-a8c2-230c028d2934\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") "
Oct 30 03:09:52.781497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.781372    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-podinfo\") pod \"a82519a5-4fac-40c4-a8c2-230c028d2934\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") "
Oct 30 03:09:52.781655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.781525    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a82519a5-4fac-40c4-a8c2-230c028d2934-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82519a5-4fac-40c4-a8c2-230c028d2934-istiod-ca-cert\") pod \"a82519a5-4fac-40c4-a8c2-230c028d2934\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") "
Oct 30 03:09:52.781655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.781578    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-socket\") pod \"a82519a5-4fac-40c4-a8c2-230c028d2934\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") "
Oct 30 03:09:52.781655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.781635    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-kube-api-access-zlvht\" (UniqueName: \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-kube-api-access-zlvht\") pod \"a82519a5-4fac-40c4-a8c2-230c028d2934\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") "
Oct 30 03:09:52.781848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.781826    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-token\" (UniqueName: \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-token\") pod \"a82519a5-4fac-40c4-a8c2-230c028d2934\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") "
Oct 30 03:09:52.782946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.782921    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-data\") pod \"a82519a5-4fac-40c4-a8c2-230c028d2934\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") "
Oct 30 03:09:52.783072 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.782969    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-certs\") pod \"a82519a5-4fac-40c4-a8c2-230c028d2934\" (UID: \"a82519a5-4fac-40c4-a8c2-230c028d2934\") "
Oct 30 03:09:52.783954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.782137    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-socket" pod "a82519a5-4fac-40c4-a8c2-230c028d2934" (UID: "a82519a5-4fac-40c4-a8c2-230c028d2934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.783954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.782451    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82519a5-4fac-40c4-a8c2-230c028d2934-istiod-ca-cert" pod "a82519a5-4fac-40c4-a8c2-230c028d2934" (UID: "a82519a5-4fac-40c4-a8c2-230c028d2934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:52.783954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.782669    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-credential-socket" pod "a82519a5-4fac-40c4-a8c2-230c028d2934" (UID: "a82519a5-4fac-40c4-a8c2-230c028d2934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.784208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.784009    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-certs" pod "a82519a5-4fac-40c4-a8c2-230c028d2934" (UID: "a82519a5-4fac-40c4-a8c2-230c028d2934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.784379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.784348    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-data" pod "a82519a5-4fac-40c4-a8c2-230c028d2934" (UID: "a82519a5-4fac-40c4-a8c2-230c028d2934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.785767 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.785730    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-envoy" pod "a82519a5-4fac-40c4-a8c2-230c028d2934" (UID: "a82519a5-4fac-40c4-a8c2-230c028d2934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.788237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.788211    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-podinfo" pod "a82519a5-4fac-40c4-a8c2-230c028d2934" (UID: "a82519a5-4fac-40c4-a8c2-230c028d2934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:52.788340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.788230    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-kube-api-access-zlvht" pod "a82519a5-4fac-40c4-a8c2-230c028d2934" (UID: "a82519a5-4fac-40c4-a8c2-230c028d2934"). InnerVolumeSpecName "kube-api-access-zlvht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:52.790900 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.790868    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-token" pod "a82519a5-4fac-40c4-a8c2-230c028d2934" (UID: "a82519a5-4fac-40c4-a8c2-230c028d2934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:52.889761 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.889722    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.890016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.889984    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82519a5-4fac-40c4-a8c2-230c028d2934-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.890016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.890016    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.890202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.890030    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlvht\" (UniqueName: \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-kube-api-access-zlvht\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.890202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.890044    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.890202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.890058    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.890202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.890071    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.890202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.890084    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.890202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.890098    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82519a5-4fac-40c4-a8c2-230c028d2934-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.935482 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:52.935408    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 03:09:53.093254 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093147    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6787da26-32e1-4016-8300-8d5f1a0143f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6787da26-32e1-4016-8300-8d5f1a0143f5-istiod-ca-cert\") pod \"6787da26-32e1-4016-8300-8d5f1a0143f5\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") "
Oct 30 03:09:53.093254 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093245    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-credential-socket\") pod \"6787da26-32e1-4016-8300-8d5f1a0143f5\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") "
Oct 30 03:09:53.093853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093373    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-kube-api-access-rw2f2\" (UniqueName: \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-kube-api-access-rw2f2\") pod \"6787da26-32e1-4016-8300-8d5f1a0143f5\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") "
Oct 30 03:09:53.093853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093453    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-envoy\") pod \"6787da26-32e1-4016-8300-8d5f1a0143f5\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") "
Oct 30 03:09:53.093853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093492    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-data\") pod \"6787da26-32e1-4016-8300-8d5f1a0143f5\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") "
Oct 30 03:09:53.093853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093552    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6787da26-32e1-4016-8300-8d5f1a0143f5-istiod-ca-cert" pod "6787da26-32e1-4016-8300-8d5f1a0143f5" (UID: "6787da26-32e1-4016-8300-8d5f1a0143f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:53.093853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093561    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-certs\") pod \"6787da26-32e1-4016-8300-8d5f1a0143f5\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") "
Oct 30 03:09:53.093853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093624    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-socket\") pod \"6787da26-32e1-4016-8300-8d5f1a0143f5\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") "
Oct 30 03:09:53.093853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093643    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-token\" (UniqueName: \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-token\") pod \"6787da26-32e1-4016-8300-8d5f1a0143f5\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") "
Oct 30 03:09:53.093853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093675    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-podinfo\") pod \"6787da26-32e1-4016-8300-8d5f1a0143f5\" (UID: \"6787da26-32e1-4016-8300-8d5f1a0143f5\") "
Oct 30 03:09:53.093853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.093836    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-certs" pod "6787da26-32e1-4016-8300-8d5f1a0143f5" (UID: "6787da26-32e1-4016-8300-8d5f1a0143f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:53.094256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.094156    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-credential-socket" pod "6787da26-32e1-4016-8300-8d5f1a0143f5" (UID: "6787da26-32e1-4016-8300-8d5f1a0143f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:53.094573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.094544    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-data" pod "6787da26-32e1-4016-8300-8d5f1a0143f5" (UID: "6787da26-32e1-4016-8300-8d5f1a0143f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:53.094753 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.094588    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-socket" pod "6787da26-32e1-4016-8300-8d5f1a0143f5" (UID: "6787da26-32e1-4016-8300-8d5f1a0143f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:53.094916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.094827    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6787da26-32e1-4016-8300-8d5f1a0143f5-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.095047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.095032    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.095148 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.095135    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.097201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.097175    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-kube-api-access-rw2f2" pod "6787da26-32e1-4016-8300-8d5f1a0143f5" (UID: "6787da26-32e1-4016-8300-8d5f1a0143f5"). InnerVolumeSpecName "kube-api-access-rw2f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:53.097201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.097178    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-envoy" pod "6787da26-32e1-4016-8300-8d5f1a0143f5" (UID: "6787da26-32e1-4016-8300-8d5f1a0143f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:53.097408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.097197    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-podinfo" pod "6787da26-32e1-4016-8300-8d5f1a0143f5" (UID: "6787da26-32e1-4016-8300-8d5f1a0143f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:53.097408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.097204    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-token" pod "6787da26-32e1-4016-8300-8d5f1a0143f5" (UID: "6787da26-32e1-4016-8300-8d5f1a0143f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:53.205467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.203040    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw2f2\" (UniqueName: \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-kube-api-access-rw2f2\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.205467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.203082    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.205467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.203100    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.205467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.203113    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6787da26-32e1-4016-8300-8d5f1a0143f5-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.205467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.203127    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.205467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.203141    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6787da26-32e1-4016-8300-8d5f1a0143f5-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.599180 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.599142    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"]
Oct 30 03:09:53.599940 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.599916    4278 cpu_manager.go:414] "containerMap: removing container" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy"
Oct 30 03:09:53.599940 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.599941    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.599966    4278 cpu_manager.go:414] "containerMap: removing container" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.599973    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600013    4278 cpu_manager.go:414] "containerMap: removing container" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600020    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600030    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerName="istio-proxy"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600037    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerName="istio-proxy"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600052    4278 cpu_manager.go:414] "containerMap: removing container" podUID="9f884cf6-bf41-45f7-837b-63b591cd2af2" containerName="revoke-keys"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600058    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f884cf6-bf41-45f7-837b-63b591cd2af2" containerName="revoke-keys"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600078    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600085    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600101    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy"
Oct 30 03:09:53.600128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600109    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy"
Oct 30 03:09:53.600713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600291    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" containerName="istio-proxy"
Oct 30 03:09:53.600713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600311    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8cc43f3-75be-43f6-8bec-59fc66f2f51f" containerName="istio-proxy"
Oct 30 03:09:53.600713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600322    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" containerName="istio-proxy"
Oct 30 03:09:53.600713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600333    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f884cf6-bf41-45f7-837b-63b591cd2af2" containerName="revoke-keys"
Oct 30 03:09:53.600713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600343    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7588026-2174-4904-a2af-8e4cdf2873f5" containerName="istio-proxy"
Oct 30 03:09:53.600713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600354    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerName="istio-proxy"
Oct 30 03:09:53.600713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.600372    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc6b7b8-98a4-46ca-b4d8-710f8a576a02" containerName="istio-proxy"
Oct 30 03:09:53.604619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.604595    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:53.624316 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.624270    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"]
Oct 30 03:09:53.631493 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.631405    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:09:53.631680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.631498    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:09:53.654320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.654287    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"
Oct 30 03:09:53.654648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.654604    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7" event={"ID":"a82519a5-4fac-40c4-a8c2-230c028d2934","Type":"ContainerDied","Data":"d86645d0dd4ab98f70abeabc14533fcc6fc05cc9bc3b0a1415d5a0a72bd5e6b6"}
Oct 30 03:09:53.654764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.654681    4278 scope.go:122] "RemoveContainer" containerID="4b48efecd1f2208f61895e4d5d67fdc8611abbc76b8bc6416231bf994a84a01b"
Oct 30 03:09:53.659554 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.659509    4278 generic.go:356] "Generic (PLEG): container finished" podID="6787da26-32e1-4016-8300-8d5f1a0143f5" containerID="0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181" exitCode=0
Oct 30 03:09:53.659784 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.659763    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" event={"ID":"6787da26-32e1-4016-8300-8d5f1a0143f5","Type":"ContainerDied","Data":"0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181"}
Oct 30 03:09:53.659870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.659826    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4" event={"ID":"6787da26-32e1-4016-8300-8d5f1a0143f5","Type":"ContainerDied","Data":"232ca9cbcb569e4533a6181842572a70b77b7c9c633edfb5544b273f9edd3af3"}
Oct 30 03:09:53.659940 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.659910    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"
Oct 30 03:09:53.674378 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.673938    4278 scope.go:122] "RemoveContainer" containerID="0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181"
Oct 30 03:09:53.684951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.684931    4278 scope.go:122] "RemoveContainer" containerID="0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181"
Oct 30 03:09:53.685230 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:09:53.685207    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181\": container with ID starting with 0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181 not found: ID does not exist" containerID="0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181"
Oct 30 03:09:53.685317 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.685244    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181"} err="failed to get container status \"0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181\": rpc error: code = NotFound desc = could not find container \"0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181\": container with ID starting with 0a7390aa078efbbcdf41e10f692c4d483c07dc87d3031498a32ba8572c0ee181 not found: ID does not exist"
Oct 30 03:09:53.718975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.718918    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-71f552-n46d6\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:53.719189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.718995    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njn9\" (UniqueName: \"kubernetes.io/projected/89e818a3-a429-4166-ab39-f99fc3edc217-kube-api-access-9njn9\") pod \"maas-api-revoke-keys-e2e-api-b-71f552-n46d6\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:53.719189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.719121    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-71f552-n46d6\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:53.720043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.720010    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"]
Oct 30 03:09:53.732350 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.732315    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-wrrv4"]
Oct 30 03:09:53.756746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.756713    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"]
Oct 30 03:09:53.765681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.765644    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qcdp7"]
Oct 30 03:09:53.820580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.820537    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-71f552-n46d6\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:53.820580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.820583    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njn9\" (UniqueName: \"kubernetes.io/projected/89e818a3-a429-4166-ab39-f99fc3edc217-kube-api-access-9njn9\") pod \"maas-api-revoke-keys-e2e-api-b-71f552-n46d6\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:53.820855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.820673    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-71f552-n46d6\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:53.821195 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.821177    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-71f552-n46d6\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:53.823082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.823058    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-71f552-n46d6\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:53.841020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.840988    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njn9\" (UniqueName: \"kubernetes.io/projected/89e818a3-a429-4166-ab39-f99fc3edc217-kube-api-access-9njn9\") pod \"maas-api-revoke-keys-e2e-api-b-71f552-n46d6\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:53.920498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:53.920340    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:54.193265 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:09:54.193183    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:09:54.203614 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:54.203583    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e529a9-14f9-4f80-96a2-2c56fd4abe64" path="/var/lib/kubelet/pods/04e529a9-14f9-4f80-96a2-2c56fd4abe64/volumes"
Oct 30 03:09:54.204190 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:54.204173    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6787da26-32e1-4016-8300-8d5f1a0143f5" path="/var/lib/kubelet/pods/6787da26-32e1-4016-8300-8d5f1a0143f5/volumes"
Oct 30 03:09:54.204656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:54.204642    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a82519a5-4fac-40c4-a8c2-230c028d2934" path="/var/lib/kubelet/pods/a82519a5-4fac-40c4-a8c2-230c028d2934/volumes"
Oct 30 03:09:54.233790 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:54.233762    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"]
Oct 30 03:09:54.235266 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:09:54.235239    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e818a3_a429_4166_ab39_f99fc3edc217.slice/crio-d68740451e3bf385d96421af1b1c165a2d9ceb49ad341ad8cd729047a7842b8b WatchSource:0}: Error finding container d68740451e3bf385d96421af1b1c165a2d9ceb49ad341ad8cd729047a7842b8b: Status 404 returned error can't find the container with id d68740451e3bf385d96421af1b1c165a2d9ceb49ad341ad8cd729047a7842b8b
Oct 30 03:09:54.670966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:54.670737    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6" event={"ID":"89e818a3-a429-4166-ab39-f99fc3edc217","Type":"ContainerStarted","Data":"d68740451e3bf385d96421af1b1c165a2d9ceb49ad341ad8cd729047a7842b8b"}
Oct 30 03:09:55.682866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:55.682759    4278 generic.go:356] "Generic (PLEG): container finished" podID="89e818a3-a429-4166-ab39-f99fc3edc217" containerID="81741fb535a3fbe8456fc3f58df99697d7de9db3130061a7bb153558d8969328" exitCode=0
Oct 30 03:09:55.682866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:55.682843    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6" event={"ID":"89e818a3-a429-4166-ab39-f99fc3edc217","Type":"ContainerDied","Data":"81741fb535a3fbe8456fc3f58df99697d7de9db3130061a7bb153558d8969328"}
Oct 30 03:09:56.861417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:56.861387    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:56.867574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:56.867547    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-xks-ca\") pod \"89e818a3-a429-4166-ab39-f99fc3edc217\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") "
Oct 30 03:09:56.867799 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:56.867784    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e818a3-a429-4166-ab39-f99fc3edc217-kube-api-access-9njn9\" (UniqueName: \"kubernetes.io/projected/89e818a3-a429-4166-ab39-f99fc3edc217-kube-api-access-9njn9\") pod \"89e818a3-a429-4166-ab39-f99fc3edc217\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") "
Oct 30 03:09:56.867857 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:56.867840    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-openshift-ca\") pod \"89e818a3-a429-4166-ab39-f99fc3edc217\" (UID: \"89e818a3-a429-4166-ab39-f99fc3edc217\") "
Oct 30 03:09:56.868245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:56.868208    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-openshift-ca" pod "89e818a3-a429-4166-ab39-f99fc3edc217" (UID: "89e818a3-a429-4166-ab39-f99fc3edc217"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:56.868484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:56.868464    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:56.870192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:56.870147    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-xks-ca" pod "89e818a3-a429-4166-ab39-f99fc3edc217" (UID: "89e818a3-a429-4166-ab39-f99fc3edc217"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:09:56.870322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:56.870241    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e818a3-a429-4166-ab39-f99fc3edc217-kube-api-access-9njn9" pod "89e818a3-a429-4166-ab39-f99fc3edc217" (UID: "89e818a3-a429-4166-ab39-f99fc3edc217"). InnerVolumeSpecName "kube-api-access-9njn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:56.969701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:56.969612    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njn9\" (UniqueName: \"kubernetes.io/projected/89e818a3-a429-4166-ab39-f99fc3edc217-kube-api-access-9njn9\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:56.969701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:56.969645    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e818a3-a429-4166-ab39-f99fc3edc217-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:09:57.700170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:57.700142    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"
Oct 30 03:09:57.700440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:57.700177    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6" event={"ID":"89e818a3-a429-4166-ab39-f99fc3edc217","Type":"ContainerDied","Data":"d68740451e3bf385d96421af1b1c165a2d9ceb49ad341ad8cd729047a7842b8b"}
Oct 30 03:09:57.700440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:09:57.700229    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d68740451e3bf385d96421af1b1c165a2d9ceb49ad341ad8cd729047a7842b8b"
Oct 30 03:10:08.515831 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.515792    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"]
Oct 30 03:10:08.516538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.516521    4278 cpu_manager.go:414] "containerMap: removing container" podUID="89e818a3-a429-4166-ab39-f99fc3edc217" containerName="revoke-keys"
Oct 30 03:10:08.516538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.516539    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e818a3-a429-4166-ab39-f99fc3edc217" containerName="revoke-keys"
Oct 30 03:10:08.516669 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.516659    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e818a3-a429-4166-ab39-f99fc3edc217" containerName="revoke-keys"
Oct 30 03:10:08.522588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.522550    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.541960 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.541924    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"]
Oct 30 03:10:08.549182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.549146    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"openshift-service-ca.crt\""
Oct 30 03:10:08.549182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.549146    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"kube-root-ca.crt\""
Oct 30 03:10:08.551520 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.551490    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"default-dockercfg-ml4hg\""
Oct 30 03:10:08.551629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.551490    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"auth-test-model-9245d1-kserve-self-signed-certs\""
Oct 30 03:10:08.607327 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.607283    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-model-cache\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.607587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.607466    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-tmp-dir\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.607587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.607510    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66096de-fc52-463a-a740-09bc1a1692d0-tls-certs\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.607691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.607638    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6ht\" (UniqueName: \"kubernetes.io/projected/f66096de-fc52-463a-a740-09bc1a1692d0-kube-api-access-fq6ht\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.607736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.607699    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-dshm\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.607837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.607810    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-home\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.709121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.709068    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6ht\" (UniqueName: \"kubernetes.io/projected/f66096de-fc52-463a-a740-09bc1a1692d0-kube-api-access-fq6ht\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.709346 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.709173    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-dshm\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.709346 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.709249    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-home\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.709489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.709403    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-model-cache\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.709551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.709494    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-tmp-dir\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.709551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.709539    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66096de-fc52-463a-a740-09bc1a1692d0-tls-certs\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.709747 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.709721    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-home\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.709849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.709827    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-model-cache\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.710015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.709992    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-tmp-dir\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.712203 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.712180    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66096de-fc52-463a-a740-09bc1a1692d0-tls-certs\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.712203 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.712192    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-dshm\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.741767 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.735478    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6ht\" (UniqueName: \"kubernetes.io/projected/f66096de-fc52-463a-a740-09bc1a1692d0-kube-api-access-fq6ht\") pod \"auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:08.856100 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:08.856004    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:09.308008 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:09.307971    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"]
Oct 30 03:10:09.308419 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:10:09.308396    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf66096de_fc52_463a_a740_09bc1a1692d0.slice/crio-2b2c531ecbc0a34e6368e2828c3a443df4379d5607ab634fe2bee9a5c3204c38 WatchSource:0}: Error finding container 2b2c531ecbc0a34e6368e2828c3a443df4379d5607ab634fe2bee9a5c3204c38: Status 404 returned error can't find the container with id 2b2c531ecbc0a34e6368e2828c3a443df4379d5607ab634fe2bee9a5c3204c38
Oct 30 03:10:09.870022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:09.869980    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4" event={"ID":"f66096de-fc52-463a-a740-09bc1a1692d0","Type":"ContainerStarted","Data":"3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5"}
Oct 30 03:10:09.870475 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:09.870034    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4" event={"ID":"f66096de-fc52-463a-a740-09bc1a1692d0","Type":"ContainerStarted","Data":"2b2c531ecbc0a34e6368e2828c3a443df4379d5607ab634fe2bee9a5c3204c38"}
Oct 30 03:10:09.911961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:09.911902    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4" podStartSLOduration=1.911886843 podStartE2EDuration="1.911886843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:09.90987197 +0000 UTC m=+1747.834080160" watchObservedRunningTime="2026-10-30 03:10:09.911886843 +0000 UTC m=+1747.836095030"
Oct 30 03:10:10.857486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:10.857445    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:16.094714 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:16.094618    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"]
Oct 30 03:10:16.107384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:16.107340    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab09ae-ffm6k"]
Oct 30 03:10:16.206299 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:16.206258    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67494687-1bad-4e8a-96a0-649422fe2b94" path="/var/lib/kubelet/pods/67494687-1bad-4e8a-96a0-649422fe2b94/volumes"
Oct 30 03:10:18.859015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:18.858977    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:18.900935 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:18.900892    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:19.016232 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:19.016200    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:10:24.323929 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:10:24.323880    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:10:27.030410 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:27.030361    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"]
Oct 30 03:10:27.043204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:27.043165    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2010e09-jbfz9"]
Oct 30 03:10:28.203993 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:28.203955    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45fbac86-8232-44c0-b41b-99bfdf0ab434" path="/var/lib/kubelet/pods/45fbac86-8232-44c0-b41b-99bfdf0ab434/volumes"
Oct 30 03:10:38.900659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.900621    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"]
Oct 30 03:10:38.905502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.905468    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:38.925641 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.925605    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"]
Oct 30 03:10:38.934309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.934265    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"kube-root-ca.crt\""
Oct 30 03:10:38.934544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.934373    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"auth-test-model-360818-kserve-self-signed-certs\""
Oct 30 03:10:38.934544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.934265    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"openshift-service-ca.crt\""
Oct 30 03:10:38.936334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.936306    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"default-dockercfg-jsdxd\""
Oct 30 03:10:38.968226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.968124    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dwtf\" (UniqueName: \"kubernetes.io/projected/80e943c9-b7ad-4d36-866d-ae0fe597fca1-kube-api-access-7dwtf\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:38.968532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.968270    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tls-certs\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:38.968532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.968461    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-model-cache\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:38.968711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.968580    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-home\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:38.968711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.968610    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tmp-dir\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:38.968711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:38.968695    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-dshm\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.074991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.073054    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-home\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.074991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.073116    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tmp-dir\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.074991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.073213    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-dshm\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.074991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.073338    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwtf\" (UniqueName: \"kubernetes.io/projected/80e943c9-b7ad-4d36-866d-ae0fe597fca1-kube-api-access-7dwtf\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.074991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.073401    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tls-certs\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.074991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.073586    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-model-cache\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.074991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.074120    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-model-cache\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.074991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.074394    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-home\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.074991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.074657    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tmp-dir\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.104113 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.092598    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tls-certs\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.104113 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.092804    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-dshm\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.121670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.121635    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dwtf\" (UniqueName: \"kubernetes.io/projected/80e943c9-b7ad-4d36-866d-ae0fe597fca1-kube-api-access-7dwtf\") pod \"auth-test-model-360818-kserve-6ffc4748f-wbpnt\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") " pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.223927 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.223739    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:39.599670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:39.599641    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"]
Oct 30 03:10:39.600334 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:10:39.600307    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80e943c9_b7ad_4d36_866d_ae0fe597fca1.slice/crio-1240733cb6f97603aa658ede1b3622bc6ed3f152b03ca3b1b3b155eda9e7d7d5 WatchSource:0}: Error finding container 1240733cb6f97603aa658ede1b3622bc6ed3f152b03ca3b1b3b155eda9e7d7d5: Status 404 returned error can't find the container with id 1240733cb6f97603aa658ede1b3622bc6ed3f152b03ca3b1b3b155eda9e7d7d5
Oct 30 03:10:40.265924 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:40.265866    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt" event={"ID":"80e943c9-b7ad-4d36-866d-ae0fe597fca1","Type":"ContainerStarted","Data":"53c6e3527fff38108492f9004b87e4d12081da31ff5211d02d3e1fac390e8829"}
Oct 30 03:10:40.266482 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:40.265936    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt" event={"ID":"80e943c9-b7ad-4d36-866d-ae0fe597fca1","Type":"ContainerStarted","Data":"1240733cb6f97603aa658ede1b3622bc6ed3f152b03ca3b1b3b155eda9e7d7d5"}
Oct 30 03:10:40.307318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:40.307247    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt" podStartSLOduration=2.307226793 podStartE2EDuration="2.307226793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:40.305223714 +0000 UTC m=+1778.229431902" watchObservedRunningTime="2026-10-30 03:10:40.307226793 +0000 UTC m=+1778.231434983"
Oct 30 03:10:41.224358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:41.224306    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:46.027828 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.027790    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"]
Oct 30 03:10:46.040281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.040243    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1671e27-892tn"]
Oct 30 03:10:46.211371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.211333    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af4ef31-b040-46f4-82a5-709fb191bf31" path="/var/lib/kubelet/pods/4af4ef31-b040-46f4-82a5-709fb191bf31/volumes"
Oct 30 03:10:46.535403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.535361    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"]
Oct 30 03:10:46.543102 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.543071    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.566494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.566458    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"]
Oct 30 03:10:46.576127 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.576095    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4f415d-openshift-default-dockercfg-qkscz\""
Oct 30 03:10:46.620448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.620379    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-data\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.620629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.620558    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-credential-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.620629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.620593    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zg8p\" (UniqueName: \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-kube-api-access-8zg8p\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.620718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.620690    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.620718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.620708    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-token\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.620797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.620779    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-envoy\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.620878 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.620862    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istiod-ca-cert\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.621037 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.621026    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-podinfo\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.621074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.621046    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-certs\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.724450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.723597    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istiod-ca-cert\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.724450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.723846    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-podinfo\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.724450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.723879    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-certs\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.724450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.723967    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-data\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.724450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.724060    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-credential-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.724450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.724101    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zg8p\" (UniqueName: \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-kube-api-access-8zg8p\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.724450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.724168    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.724450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.724196    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-token\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.724450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.724246    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-envoy\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.749451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.725293    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-data\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.749451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.725758    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-certs\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.749451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.725954    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istiod-ca-cert\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.749451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.726012    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-credential-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.749451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.726186    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.778450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.760062    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-envoy\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.778450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.760567    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-podinfo\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.796053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.795972    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-token\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.811457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.803088    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zg8p\" (UniqueName: \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-kube-api-access-8zg8p\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:46.870901 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:46.870520    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:47.287335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:47.287303    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"]
Oct 30 03:10:47.288580 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:10:47.288552    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffbabd1_d7b2_4b61_aa48_58b0be40e9da.slice/crio-6e468d07e492f3284e41265fefb6ceb4e560ca00ab60071674c64e913b0b9c9f WatchSource:0}: Error finding container 6e468d07e492f3284e41265fefb6ceb4e560ca00ab60071674c64e913b0b9c9f: Status 404 returned error can't find the container with id 6e468d07e492f3284e41265fefb6ceb4e560ca00ab60071674c64e913b0b9c9f
Oct 30 03:10:47.291843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:47.291809    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:47.291933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:47.291880    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:47.291933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:47.291912    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:47.333584 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:47.330392    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" event={"ID":"dffbabd1-d7b2-4b61-aa48-58b0be40e9da","Type":"ContainerStarted","Data":"6e468d07e492f3284e41265fefb6ceb4e560ca00ab60071674c64e913b0b9c9f"}
Oct 30 03:10:48.341984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:48.341942    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" event={"ID":"dffbabd1-d7b2-4b61-aa48-58b0be40e9da","Type":"ContainerStarted","Data":"ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb"}
Oct 30 03:10:48.414723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:48.414653    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" podStartSLOduration=2.414628034 podStartE2EDuration="2.414628034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:48.398306604 +0000 UTC m=+1786.322514792" watchObservedRunningTime="2026-10-30 03:10:48.414628034 +0000 UTC m=+1786.338836225"
Oct 30 03:10:48.872255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:48.872219    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:48.879229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:48.879197    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:49.224984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.224889    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:49.238702 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.238669    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:49.350320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.350281    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:49.351305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.351281    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:10:49.361616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.361592    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:10:49.458785 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.458744    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"]
Oct 30 03:10:49.466242 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.466210    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.479944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.479871    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"]
Oct 30 03:10:49.512712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.506179    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4cgc\" (UniqueName: \"kubernetes.io/projected/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-kube-api-access-j4cgc\") pod \"maas-api-revoke-sub-b1991a36-92t7v\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.512712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.506266    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1991a36-92t7v\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.512712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.506328    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1991a36-92t7v\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.564778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.564744    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:10:49.564778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.564771    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:10:49.607363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.607324    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1991a36-92t7v\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.607646 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.607380    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1991a36-92t7v\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.607776 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.607757    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4cgc\" (UniqueName: \"kubernetes.io/projected/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-kube-api-access-j4cgc\") pod \"maas-api-revoke-sub-b1991a36-92t7v\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.608173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.608146    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1991a36-92t7v\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.609985 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.609967    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1991a36-92t7v\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.627969 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.627943    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4cgc\" (UniqueName: \"kubernetes.io/projected/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-kube-api-access-j4cgc\") pod \"maas-api-revoke-sub-b1991a36-92t7v\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.791108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.791069    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:49.891657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.891615    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"]
Oct 30 03:10:49.896669 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.896642    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:49.917718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.917674    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv5z7\" (UniqueName: \"kubernetes.io/projected/7d0c0134-d2ad-4571-b491-fc8afe92a567-kube-api-access-bv5z7\") pod \"maas-api-revoke-keys-e2e-api-a-67440d-v5ckv\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:49.917943 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.917802    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-67440d-v5ckv\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:49.918022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.917938    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-67440d-v5ckv\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:49.925385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:49.925342    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"]
Oct 30 03:10:50.020005 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:50.019965    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv5z7\" (UniqueName: \"kubernetes.io/projected/7d0c0134-d2ad-4571-b491-fc8afe92a567-kube-api-access-bv5z7\") pod \"maas-api-revoke-keys-e2e-api-a-67440d-v5ckv\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:50.020198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:50.020014    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-67440d-v5ckv\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:50.020198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:50.020059    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-67440d-v5ckv\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:50.020929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:50.020905    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-67440d-v5ckv\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:50.022566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:50.022535    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-67440d-v5ckv\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:50.039883 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:50.039851    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv5z7\" (UniqueName: \"kubernetes.io/projected/7d0c0134-d2ad-4571-b491-fc8afe92a567-kube-api-access-bv5z7\") pod \"maas-api-revoke-keys-e2e-api-a-67440d-v5ckv\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:50.107709 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:50.104494    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"]
Oct 30 03:10:50.228418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:50.228381    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:50.368112 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:50.368011    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v" event={"ID":"0242ff22-6f8a-4f8f-a820-5e6ab21378ef","Type":"ContainerStarted","Data":"75c0aa5b7f3af2c084eff242febbf591292fcf9ab40360a2ba4b540a9844b11d"}
Oct 30 03:10:50.539753 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:10:50.539720    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0c0134_d2ad_4571_b491_fc8afe92a567.slice/crio-beff0f72819a3612a04f407f92352f6f8759cca1366f862ba1752a6381d1dbc5 WatchSource:0}: Error finding container beff0f72819a3612a04f407f92352f6f8759cca1366f862ba1752a6381d1dbc5: Status 404 returned error can't find the container with id beff0f72819a3612a04f407f92352f6f8759cca1366f862ba1752a6381d1dbc5
Oct 30 03:10:50.541231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:50.541205    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"]
Oct 30 03:10:51.377750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:51.377642    4278 generic.go:356] "Generic (PLEG): container finished" podID="0242ff22-6f8a-4f8f-a820-5e6ab21378ef" containerID="54832ee542fa59bfdef13dbd60bfc936f7f55db7e1820e082cfbbc85190533bb" exitCode=0
Oct 30 03:10:51.377750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:51.377729    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v" event={"ID":"0242ff22-6f8a-4f8f-a820-5e6ab21378ef","Type":"ContainerDied","Data":"54832ee542fa59bfdef13dbd60bfc936f7f55db7e1820e082cfbbc85190533bb"}
Oct 30 03:10:51.379488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:51.379462    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv" event={"ID":"7d0c0134-d2ad-4571-b491-fc8afe92a567","Type":"ContainerStarted","Data":"c238435e4ee2a1de1e5784aa3d41ac12ece75cd12c1b569ad3e5844dc0bc93cb"}
Oct 30 03:10:51.379629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:51.379507    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv" event={"ID":"7d0c0134-d2ad-4571-b491-fc8afe92a567","Type":"ContainerStarted","Data":"beff0f72819a3612a04f407f92352f6f8759cca1366f862ba1752a6381d1dbc5"}
Oct 30 03:10:51.511127 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:51.511065    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv" podStartSLOduration=2.511044746 podStartE2EDuration="2.511044746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:51.510381883 +0000 UTC m=+1789.434590099" watchObservedRunningTime="2026-10-30 03:10:51.511044746 +0000 UTC m=+1789.435252939"
Oct 30 03:10:52.389269 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.389230    4278 generic.go:356] "Generic (PLEG): container finished" podID="7d0c0134-d2ad-4571-b491-fc8afe92a567" containerID="c238435e4ee2a1de1e5784aa3d41ac12ece75cd12c1b569ad3e5844dc0bc93cb" exitCode=0
Oct 30 03:10:52.389735 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.389317    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv" event={"ID":"7d0c0134-d2ad-4571-b491-fc8afe92a567","Type":"ContainerDied","Data":"c238435e4ee2a1de1e5784aa3d41ac12ece75cd12c1b569ad3e5844dc0bc93cb"}
Oct 30 03:10:52.585879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.585844    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:52.665290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.665198    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-kube-api-access-j4cgc\" (UniqueName: \"kubernetes.io/projected/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-kube-api-access-j4cgc\") pod \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") "
Oct 30 03:10:52.665290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.665248    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-xks-ca\") pod \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") "
Oct 30 03:10:52.665569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.665438    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-openshift-ca\") pod \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\" (UID: \"0242ff22-6f8a-4f8f-a820-5e6ab21378ef\") "
Oct 30 03:10:52.665898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.665868    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-openshift-ca" pod "0242ff22-6f8a-4f8f-a820-5e6ab21378ef" (UID: "0242ff22-6f8a-4f8f-a820-5e6ab21378ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:10:52.666261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.666235    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:10:52.669883 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.669847    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-kube-api-access-j4cgc" pod "0242ff22-6f8a-4f8f-a820-5e6ab21378ef" (UID: "0242ff22-6f8a-4f8f-a820-5e6ab21378ef"). InnerVolumeSpecName "kube-api-access-j4cgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:10:52.669989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.669902    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-xks-ca" pod "0242ff22-6f8a-4f8f-a820-5e6ab21378ef" (UID: "0242ff22-6f8a-4f8f-a820-5e6ab21378ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:10:52.774545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.774500    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4cgc\" (UniqueName: \"kubernetes.io/projected/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-kube-api-access-j4cgc\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:10:52.774826 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:52.774795    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0242ff22-6f8a-4f8f-a820-5e6ab21378ef-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:10:53.399148 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.399116    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"
Oct 30 03:10:53.399704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.399146    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v" event={"ID":"0242ff22-6f8a-4f8f-a820-5e6ab21378ef","Type":"ContainerDied","Data":"75c0aa5b7f3af2c084eff242febbf591292fcf9ab40360a2ba4b540a9844b11d"}
Oct 30 03:10:53.399704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.399197    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75c0aa5b7f3af2c084eff242febbf591292fcf9ab40360a2ba4b540a9844b11d"
Oct 30 03:10:53.592396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.592368    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:53.713676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.713585    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0c0134-d2ad-4571-b491-fc8afe92a567-kube-api-access-bv5z7\" (UniqueName: \"kubernetes.io/projected/7d0c0134-d2ad-4571-b491-fc8afe92a567-kube-api-access-bv5z7\") pod \"7d0c0134-d2ad-4571-b491-fc8afe92a567\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") "
Oct 30 03:10:53.713948 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.713925    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-xks-ca\") pod \"7d0c0134-d2ad-4571-b491-fc8afe92a567\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") "
Oct 30 03:10:53.714037 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.714022    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-openshift-ca\") pod \"7d0c0134-d2ad-4571-b491-fc8afe92a567\" (UID: \"7d0c0134-d2ad-4571-b491-fc8afe92a567\") "
Oct 30 03:10:53.714386 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.714356    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-openshift-ca" pod "7d0c0134-d2ad-4571-b491-fc8afe92a567" (UID: "7d0c0134-d2ad-4571-b491-fc8afe92a567"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:10:53.714731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.714707    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:10:53.715976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.715947    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-xks-ca" pod "7d0c0134-d2ad-4571-b491-fc8afe92a567" (UID: "7d0c0134-d2ad-4571-b491-fc8afe92a567"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:10:53.716077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.715973    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0c0134-d2ad-4571-b491-fc8afe92a567-kube-api-access-bv5z7" pod "7d0c0134-d2ad-4571-b491-fc8afe92a567" (UID: "7d0c0134-d2ad-4571-b491-fc8afe92a567"). InnerVolumeSpecName "kube-api-access-bv5z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:10:53.816521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.816399    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d0c0134-d2ad-4571-b491-fc8afe92a567-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:10:53.816521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:53.816518    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv5z7\" (UniqueName: \"kubernetes.io/projected/7d0c0134-d2ad-4571-b491-fc8afe92a567-kube-api-access-bv5z7\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:10:54.192836 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:10:54.192721    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:10:54.409182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:54.409133    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv" event={"ID":"7d0c0134-d2ad-4571-b491-fc8afe92a567","Type":"ContainerDied","Data":"beff0f72819a3612a04f407f92352f6f8759cca1366f862ba1752a6381d1dbc5"}
Oct 30 03:10:54.409182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:54.409164    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"
Oct 30 03:10:54.409745 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:10:54.409193    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beff0f72819a3612a04f407f92352f6f8759cca1366f862ba1752a6381d1dbc5"
Oct 30 03:11:10.950397 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.950357    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"]
Oct 30 03:11:10.951235 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.951210    4278 cpu_manager.go:414] "containerMap: removing container" podUID="7d0c0134-d2ad-4571-b491-fc8afe92a567" containerName="revoke-keys"
Oct 30 03:11:10.951235 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.951235    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0c0134-d2ad-4571-b491-fc8afe92a567" containerName="revoke-keys"
Oct 30 03:11:10.951458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.951297    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0242ff22-6f8a-4f8f-a820-5e6ab21378ef" containerName="revoke-keys"
Oct 30 03:11:10.951458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.951306    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0242ff22-6f8a-4f8f-a820-5e6ab21378ef" containerName="revoke-keys"
Oct 30 03:11:10.951578 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.951501    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0242ff22-6f8a-4f8f-a820-5e6ab21378ef" containerName="revoke-keys"
Oct 30 03:11:10.951578 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.951533    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0c0134-d2ad-4571-b491-fc8afe92a567" containerName="revoke-keys"
Oct 30 03:11:10.956958 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.956937    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:10.979398 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.979368    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"]
Oct 30 03:11:10.995963 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.995932    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c990b694-tffjs\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:10.996352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.996323    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2g6m\" (UniqueName: \"kubernetes.io/projected/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-kube-api-access-d2g6m\") pod \"maas-api-revoke-sub-c990b694-tffjs\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:10.996419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:10.996401    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c990b694-tffjs\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:11.051362 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:11.051327    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:11:11.051610 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:11.051393    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:11:11.098445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:11.098387    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2g6m\" (UniqueName: \"kubernetes.io/projected/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-kube-api-access-d2g6m\") pod \"maas-api-revoke-sub-c990b694-tffjs\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:11.098711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:11.098495    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c990b694-tffjs\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:11.098711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:11.098669    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c990b694-tffjs\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:11.099302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:11.099276    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c990b694-tffjs\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:11.101627 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:11.101597    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c990b694-tffjs\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:11.119489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:11.119460    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2g6m\" (UniqueName: \"kubernetes.io/projected/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-kube-api-access-d2g6m\") pod \"maas-api-revoke-sub-c990b694-tffjs\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:11.272534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:11.272488    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:11.582547 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:11:11.582509    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bad01a_cef3_4d53_be60_2e2e0de1fc8e.slice/crio-05c8ac39b7e31244749b55907e8ce667b7dec74a10f4fc30f4e2fc565bf03ffa WatchSource:0}: Error finding container 05c8ac39b7e31244749b55907e8ce667b7dec74a10f4fc30f4e2fc565bf03ffa: Status 404 returned error can't find the container with id 05c8ac39b7e31244749b55907e8ce667b7dec74a10f4fc30f4e2fc565bf03ffa
Oct 30 03:11:11.583413 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:11.583388    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"]
Oct 30 03:11:12.591324 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:12.591286    4278 generic.go:356] "Generic (PLEG): container finished" podID="25bad01a-cef3-4d53-be60-2e2e0de1fc8e" containerID="20031042400e856d446ee1bdcc53d4d2d275392dce13911974cda4afbca2d2a6" exitCode=0
Oct 30 03:11:12.591758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:12.591362    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs" event={"ID":"25bad01a-cef3-4d53-be60-2e2e0de1fc8e","Type":"ContainerDied","Data":"20031042400e856d446ee1bdcc53d4d2d275392dce13911974cda4afbca2d2a6"}
Oct 30 03:11:12.591758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:12.591444    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs" event={"ID":"25bad01a-cef3-4d53-be60-2e2e0de1fc8e","Type":"ContainerStarted","Data":"05c8ac39b7e31244749b55907e8ce667b7dec74a10f4fc30f4e2fc565bf03ffa"}
Oct 30 03:11:13.767611 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:13.767579    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:13.859104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:13.858997    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-openshift-ca\") pod \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") "
Oct 30 03:11:13.859300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:13.859236    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-kube-api-access-d2g6m\" (UniqueName: \"kubernetes.io/projected/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-kube-api-access-d2g6m\") pod \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") "
Oct 30 03:11:13.859472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:13.859454    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-xks-ca\") pod \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\" (UID: \"25bad01a-cef3-4d53-be60-2e2e0de1fc8e\") "
Oct 30 03:11:13.859598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:13.859568    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-openshift-ca" pod "25bad01a-cef3-4d53-be60-2e2e0de1fc8e" (UID: "25bad01a-cef3-4d53-be60-2e2e0de1fc8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:13.860121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:13.860096    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:11:13.861670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:13.861640    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-kube-api-access-d2g6m" pod "25bad01a-cef3-4d53-be60-2e2e0de1fc8e" (UID: "25bad01a-cef3-4d53-be60-2e2e0de1fc8e"). InnerVolumeSpecName "kube-api-access-d2g6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:13.862872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:13.862847    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-xks-ca" pod "25bad01a-cef3-4d53-be60-2e2e0de1fc8e" (UID: "25bad01a-cef3-4d53-be60-2e2e0de1fc8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:13.962040 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:13.961989    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2g6m\" (UniqueName: \"kubernetes.io/projected/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-kube-api-access-d2g6m\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:11:13.962040 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:13.962032    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bad01a-cef3-4d53-be60-2e2e0de1fc8e-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:11:14.608053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:14.608021    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"
Oct 30 03:11:14.608243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:14.608019    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs" event={"ID":"25bad01a-cef3-4d53-be60-2e2e0de1fc8e","Type":"ContainerDied","Data":"05c8ac39b7e31244749b55907e8ce667b7dec74a10f4fc30f4e2fc565bf03ffa"}
Oct 30 03:11:14.608243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:14.608151    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05c8ac39b7e31244749b55907e8ce667b7dec74a10f4fc30f4e2fc565bf03ffa"
Oct 30 03:11:15.406369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:15.406329    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:11:17.295156 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:17.295116    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"]
Oct 30 03:11:17.307150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:17.307114    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-71f552-n46d6"]
Oct 30 03:11:18.203359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:18.203322    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e818a3-a429-4166-ab39-f99fc3edc217" path="/var/lib/kubelet/pods/89e818a3-a429-4166-ab39-f99fc3edc217/volumes"
Oct 30 03:11:24.192818 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:11:24.192767    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:11:40.894055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.894012    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"]
Oct 30 03:11:40.895213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.895172    4278 cpu_manager.go:414] "containerMap: removing container" podUID="25bad01a-cef3-4d53-be60-2e2e0de1fc8e" containerName="revoke-keys"
Oct 30 03:11:40.895362 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.895221    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25bad01a-cef3-4d53-be60-2e2e0de1fc8e" containerName="revoke-keys"
Oct 30 03:11:40.895560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.895537    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="25bad01a-cef3-4d53-be60-2e2e0de1fc8e" containerName="revoke-keys"
Oct 30 03:11:40.899986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.899957    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:40.926109 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.926017    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"]
Oct 30 03:11:40.959492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.953528    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:40.959853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.959824    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8z7j\" (UniqueName: \"kubernetes.io/projected/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-kube-api-access-l8z7j\") pod \"maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:40.960036 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.960016    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:40.996916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.996756    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:11:40.996916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:40.996886    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:11:41.064366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:41.064326    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:41.064629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:41.064380    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8z7j\" (UniqueName: \"kubernetes.io/projected/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-kube-api-access-l8z7j\") pod \"maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:41.064629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:41.064411    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:41.065267 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:41.065238    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:41.067592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:41.067559    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:41.097316 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:41.097282    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8z7j\" (UniqueName: \"kubernetes.io/projected/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-kube-api-access-l8z7j\") pod \"maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:41.229566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:41.229406    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:41.592004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:41.591970    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"]
Oct 30 03:11:41.593301 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:11:41.593270    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb0aee2_6472_46b9_9c0b_f13111e7f6f6.slice/crio-66270b84db42fa371ab130d7e102e32d013d9037bb8e04f00445f817ca84f58e WatchSource:0}: Error finding container 66270b84db42fa371ab130d7e102e32d013d9037bb8e04f00445f817ca84f58e: Status 404 returned error can't find the container with id 66270b84db42fa371ab130d7e102e32d013d9037bb8e04f00445f817ca84f58e
Oct 30 03:11:41.890501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:41.890376    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b" event={"ID":"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6","Type":"ContainerStarted","Data":"66270b84db42fa371ab130d7e102e32d013d9037bb8e04f00445f817ca84f58e"}
Oct 30 03:11:42.900155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:42.900059    4278 generic.go:356] "Generic (PLEG): container finished" podID="9fb0aee2-6472-46b9-9c0b-f13111e7f6f6" containerID="d1fc6b9567d22822ed102b5a8c30581e66ef80ae4f72898e0c48a50311efb95b" exitCode=0
Oct 30 03:11:42.900630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:42.900145    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b" event={"ID":"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6","Type":"ContainerDied","Data":"d1fc6b9567d22822ed102b5a8c30581e66ef80ae4f72898e0c48a50311efb95b"}
Oct 30 03:11:44.087136 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.087102    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:44.219116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.219029    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-openshift-ca\") pod \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") "
Oct 30 03:11:44.219280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.219127    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-xks-ca\") pod \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") "
Oct 30 03:11:44.219388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.219365    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-kube-api-access-l8z7j\" (UniqueName: \"kubernetes.io/projected/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-kube-api-access-l8z7j\") pod \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\" (UID: \"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6\") "
Oct 30 03:11:44.219517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.219471    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-openshift-ca" pod "9fb0aee2-6472-46b9-9c0b-f13111e7f6f6" (UID: "9fb0aee2-6472-46b9-9c0b-f13111e7f6f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:44.220058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.220035    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:11:44.221497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.221472    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-xks-ca" pod "9fb0aee2-6472-46b9-9c0b-f13111e7f6f6" (UID: "9fb0aee2-6472-46b9-9c0b-f13111e7f6f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:44.221588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.221560    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-kube-api-access-l8z7j" pod "9fb0aee2-6472-46b9-9c0b-f13111e7f6f6" (UID: "9fb0aee2-6472-46b9-9c0b-f13111e7f6f6"). InnerVolumeSpecName "kube-api-access-l8z7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:44.321613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.321561    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:11:44.321613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.321606    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8z7j\" (UniqueName: \"kubernetes.io/projected/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6-kube-api-access-l8z7j\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:11:44.917904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.917821    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"
Oct 30 03:11:44.917904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.917849    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b" event={"ID":"9fb0aee2-6472-46b9-9c0b-f13111e7f6f6","Type":"ContainerDied","Data":"66270b84db42fa371ab130d7e102e32d013d9037bb8e04f00445f817ca84f58e"}
Oct 30 03:11:44.918111 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:44.917911    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66270b84db42fa371ab130d7e102e32d013d9037bb8e04f00445f817ca84f58e"
Oct 30 03:11:54.190751 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:11:54.190706    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:11:55.905029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.904994    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"]
Oct 30 03:11:55.905700 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.905682    4278 cpu_manager.go:414] "containerMap: removing container" podUID="9fb0aee2-6472-46b9-9c0b-f13111e7f6f6" containerName="revoke-keys"
Oct 30 03:11:55.905753 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.905701    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb0aee2-6472-46b9-9c0b-f13111e7f6f6" containerName="revoke-keys"
Oct 30 03:11:55.905840 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.905831    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb0aee2-6472-46b9-9c0b-f13111e7f6f6" containerName="revoke-keys"
Oct 30 03:11:55.909339 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.909317    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:55.924445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.924406    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"]
Oct 30 03:11:55.928829 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.928810    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:55.932029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.932011    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9245d1-openshift-default-dockercfg-884td\""
Oct 30 03:11:55.941516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.941492    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"]
Oct 30 03:11:55.960539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.959155    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"]
Oct 30 03:11:55.969757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.969712    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:55.969929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.969885    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:55.969994 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.969953    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:55.970104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.970080    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:55.970141 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.970122    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e31519d-e465-4414-b22b-554f662ae87f-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:55.970206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.970192    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e31519d-e465-4414-b22b-554f662ae87f-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:55.970259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.970244    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkn7\" (UniqueName: \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-kube-api-access-mtkn7\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:55.970346 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.970332    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:55.970388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:55.970374    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.071713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.071678    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.071713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.071722    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.071972 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.071781    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4748216d-6793-4ea1-8bc8-e6982cb82707-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.071972 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.071819    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.072083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072022    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.072194 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072078    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072194 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072101    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072197    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.072291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072236    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072329    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072410    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072457    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e31519d-e465-4414-b22b-554f662ae87f-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072466    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072501    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.072663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072548    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.072663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072626    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e31519d-e465-4414-b22b-554f662ae87f-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072677    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmd4w\" (UniqueName: \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-kube-api-access-rmd4w\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.072798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072678    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072727    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.072798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072735    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072807    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkn7\" (UniqueName: \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-kube-api-access-mtkn7\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.072925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.072864    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.073127 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.073109    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e31519d-e465-4414-b22b-554f662ae87f-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.074149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.074132    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.075108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.075090    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e31519d-e465-4414-b22b-554f662ae87f-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.093382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.093350    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkn7\" (UniqueName: \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-kube-api-access-mtkn7\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.093612 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.093590    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.174141 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174050    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174141 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174094    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174141 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174130    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmd4w\" (UniqueName: \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-kube-api-access-rmd4w\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174150    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174188    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174270    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4748216d-6793-4ea1-8bc8-e6982cb82707-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174337    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174396    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174522    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174654    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174710    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174930 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174841    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.174930 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.174908    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.175083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.175064    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4748216d-6793-4ea1-8bc8-e6982cb82707-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.176862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.176837    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.176977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.176893    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.194576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.194543    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.194687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.194557    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmd4w\" (UniqueName: \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-kube-api-access-rmd4w\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.225876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.225843    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:56.268713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.259744    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:56.748137 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.748054    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"]
Oct 30 03:11:56.760597 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:11:56.760565    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4748216d_6793_4ea1_8bc8_e6982cb82707.slice/crio-8428e76a3e88302626932c050c34dcee5d0ba03c8582f7ed4f94cf51f0bb6d84 WatchSource:0}: Error finding container 8428e76a3e88302626932c050c34dcee5d0ba03c8582f7ed4f94cf51f0bb6d84: Status 404 returned error can't find the container with id 8428e76a3e88302626932c050c34dcee5d0ba03c8582f7ed4f94cf51f0bb6d84
Oct 30 03:11:56.763765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.763732    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:56.763862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.763807    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:56.763862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.763839    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:56.787047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.787016    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"]
Oct 30 03:11:56.787488 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:11:56.787461    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e31519d_e465_4414_b22b_554f662ae87f.slice/crio-6b868d2809c5e6c4606d5a25d9493805c6b35967879d1a1304a6476313b950a5 WatchSource:0}: Error finding container 6b868d2809c5e6c4606d5a25d9493805c6b35967879d1a1304a6476313b950a5: Status 404 returned error can't find the container with id 6b868d2809c5e6c4606d5a25d9493805c6b35967879d1a1304a6476313b950a5
Oct 30 03:11:56.790396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.790359    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:56.790536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.790459    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:56.790536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:56.790510    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:57.079802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.079752    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" event={"ID":"4748216d-6793-4ea1-8bc8-e6982cb82707","Type":"ContainerStarted","Data":"d9388cd3431f23ba24d2037dab3aabb6e370f7a449b01bfb8d3f56e4ecd3bea5"}
Oct 30 03:11:57.080292 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.079827    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" event={"ID":"4748216d-6793-4ea1-8bc8-e6982cb82707","Type":"ContainerStarted","Data":"8428e76a3e88302626932c050c34dcee5d0ba03c8582f7ed4f94cf51f0bb6d84"}
Oct 30 03:11:57.082531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.082490    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" event={"ID":"9e31519d-e465-4414-b22b-554f662ae87f","Type":"ContainerStarted","Data":"77b5b5a1aacf1ff2167ed1a2d3c3d2965c9e23debc9d91ce56d2274304e3766d"}
Oct 30 03:11:57.082691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.082546    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" event={"ID":"9e31519d-e465-4414-b22b-554f662ae87f","Type":"ContainerStarted","Data":"6b868d2809c5e6c4606d5a25d9493805c6b35967879d1a1304a6476313b950a5"}
Oct 30 03:11:57.134412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.134342    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" podStartSLOduration=2.134322396 podStartE2EDuration="2.134322396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:57.128042869 +0000 UTC m=+1855.052251057" watchObservedRunningTime="2026-10-30 03:11:57.134322396 +0000 UTC m=+1855.058530584"
Oct 30 03:11:57.183757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.183700    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" podStartSLOduration=2.183683622 podStartE2EDuration="2.183683622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:57.178096488 +0000 UTC m=+1855.102304678" watchObservedRunningTime="2026-10-30 03:11:57.183683622 +0000 UTC m=+1855.107891810"
Oct 30 03:11:57.227193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.227149    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:11:57.228757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.228729    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:11:57.228904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.228776    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 03:11:57.262275 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.262225    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:11:57.265487 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.265406    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:11:57.265660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:57.265496    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 03:11:58.226783 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:58.226738    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:11:58.227290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:58.226804    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 03:11:58.300448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:58.300156    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:11:58.300448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:58.300222    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 03:11:59.227392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:59.227355    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:11:59.227956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:59.227417    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 03:11:59.268813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:59.268775    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:11:59.269106 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:11:59.269080    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 03:12:00.226516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:00.226467    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:12:00.226764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:00.226534    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 03:12:00.279451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:00.267726    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:12:00.279451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:00.267787    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 03:12:01.233622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:01.233153    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:12:01.233622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:01.233220    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 03:12:01.277628 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:01.277584    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:12:01.277945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:01.277912    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 03:12:02.229150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:02.229106    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:12:02.229654 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:02.229183    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 03:12:02.267355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:02.267316    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:12:02.284450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:02.267646    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 03:12:03.248501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:03.248240    4278 scope.go:122] "RemoveContainer" containerID="5146a2b93925e7ae8e3916f9f21a1fb74363543452e2e4b5d5bec06e3f094587"
Oct 30 03:12:03.254747 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:03.254707    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:12:03.255050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:03.255013    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 03:12:03.273210 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:03.273166    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:12:03.277066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:03.274598    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 03:12:03.318221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:03.318195    4278 scope.go:122] "RemoveContainer" containerID="7597080b1f5975e6363bba9f5c3782142a61c936b5583adf3d13d5d7ce41f50a"
Oct 30 03:12:03.330582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:03.330548    4278 scope.go:122] "RemoveContainer" containerID="542ac6c44ef1fd890d4c9d2fd9dc42ee2a54a674f0e668b57ea565ed9df295cb"
Oct 30 03:12:04.232437 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:04.232392    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:12:04.232768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:04.232750    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:12:04.233568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:04.233550    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:12:04.267904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:04.267876    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:12:04.268292 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:04.268272    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:12:04.269054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:04.269034    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:12:07.976063 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:07.976029    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"]
Oct 30 03:12:07.989506 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:07.989475    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-67440d-v5ckv"]
Oct 30 03:12:08.201951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:08.201920    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0c0134-d2ad-4571-b491-fc8afe92a567" path="/var/lib/kubelet/pods/7d0c0134-d2ad-4571-b491-fc8afe92a567/volumes"
Oct 30 03:12:09.030782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:09.030744    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"]
Oct 30 03:12:09.042903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:09.042863    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866f6ca6-f8j2p"]
Oct 30 03:12:10.228149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:10.228106    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3faf8cf9-2bb5-4863-9740-b907ced8f2cd" path="/var/lib/kubelet/pods/3faf8cf9-2bb5-4863-9740-b907ced8f2cd/volumes"
Oct 30 03:12:11.005394 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.005357    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"]
Oct 30 03:12:11.013360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.013330    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.050068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.050033    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"]
Oct 30 03:12:11.102492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.102460    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.102665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.102506    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwl8f\" (UniqueName: \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-kube-api-access-lwl8f\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.102665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.102577    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.102755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.102684    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.102755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.102726    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.102912 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.102886    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.102973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.102930    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.103047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.103029    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.103132 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.103115    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205114    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205339 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205298    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205441 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205343    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwl8f\" (UniqueName: \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-kube-api-access-lwl8f\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205441 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205404    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205488    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205549    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205609    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205675    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205693    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205727    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205911 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205772    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.205965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.205919    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.206231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.206194    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.206363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.206344    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.208281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.208252    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.208506 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.208485    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.231488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.231456    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwl8f\" (UniqueName: \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-kube-api-access-lwl8f\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.231898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.231522    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.342381 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.342342    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:11.964464 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:12:11.964407    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137707a1_f8b6_4f13_8eb3_987eb1284ad6.slice/crio-c102e42d09f614e658a943d9e4ce53948cfb5f82c5e29ac663c58e4e49bdfb83 WatchSource:0}: Error finding container c102e42d09f614e658a943d9e4ce53948cfb5f82c5e29ac663c58e4e49bdfb83: Status 404 returned error can't find the container with id c102e42d09f614e658a943d9e4ce53948cfb5f82c5e29ac663c58e4e49bdfb83
Oct 30 03:12:11.965556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.965522    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"]
Oct 30 03:12:11.967797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.967760    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:11.967887 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.967839    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:11.967887 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:11.967878    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:12.377523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.377488    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"]
Oct 30 03:12:12.382512 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.382343    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.409887 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.409855    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-16df10\"/\"test-model-16df10-kserve-self-signed-certs\""
Oct 30 03:12:12.422914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.422883    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-16df10\"/\"default-dockercfg-4w9dq\""
Oct 30 03:12:12.423095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.423070    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-16df10\"/\"openshift-service-ca.crt\""
Oct 30 03:12:12.423283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.423233    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-16df10\"/\"kube-root-ca.crt\""
Oct 30 03:12:12.427782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.427756    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"]
Oct 30 03:12:12.432794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.432770    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919c258f-20ff-488a-a62e-9bc61dcd5319-tls-certs\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.432922 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.432803    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-home\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.432991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.432973    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-tmp-dir\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.433156 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.433137    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-model-cache\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.433276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.433173    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qzp\" (UniqueName: \"kubernetes.io/projected/919c258f-20ff-488a-a62e-9bc61dcd5319-kube-api-access-d6qzp\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.433276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.433260    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-dshm\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.499860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.499823    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" event={"ID":"137707a1-f8b6-4f13-8eb3-987eb1284ad6","Type":"ContainerStarted","Data":"5fbfebaa37d99c74631dcc3cd48527b9cb7b4ed64a8f05e146e821e94bbb766f"}
Oct 30 03:12:12.500054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.499912    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" event={"ID":"137707a1-f8b6-4f13-8eb3-987eb1284ad6","Type":"ContainerStarted","Data":"c102e42d09f614e658a943d9e4ce53948cfb5f82c5e29ac663c58e4e49bdfb83"}
Oct 30 03:12:12.534442 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.534387    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-dshm\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.534617 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.534530    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919c258f-20ff-488a-a62e-9bc61dcd5319-tls-certs\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.534617 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.534560    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-home\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.534773 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.534737    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-tmp-dir\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.535459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.534969    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-model-cache\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.535459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.534998    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-home\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.535459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.535016    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qzp\" (UniqueName: \"kubernetes.io/projected/919c258f-20ff-488a-a62e-9bc61dcd5319-kube-api-access-d6qzp\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.535459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.535321    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-tmp-dir\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.535459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.535382    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-model-cache\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.537609 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.537584    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-dshm\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.537864 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.537848    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919c258f-20ff-488a-a62e-9bc61dcd5319-tls-certs\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.547333 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.547216    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" podStartSLOduration=2.5472016650000002 podStartE2EDuration="2.547201665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:12.546657541 +0000 UTC m=+1870.470865731" watchObservedRunningTime="2026-10-30 03:12:12.547201665 +0000 UTC m=+1870.471409854"
Oct 30 03:12:12.584445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.582304    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qzp\" (UniqueName: \"kubernetes.io/projected/919c258f-20ff-488a-a62e-9bc61dcd5319-kube-api-access-d6qzp\") pod \"test-model-16df10-kserve-794dfff7bc-lj2sx\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") " pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:12.717950 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:12.716786    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:13.340747 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:13.339927    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"]
Oct 30 03:12:13.343496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:13.343243    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:13.345266 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:13.345178    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 03:12:13.345440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:13.345264    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 03:12:13.511457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:13.511318    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx" event={"ID":"919c258f-20ff-488a-a62e-9bc61dcd5319","Type":"ContainerStarted","Data":"bafd7ac7d13bf9c26341cb8d60b52e30e1afc1efdd38a6b6e8eb9d023f99a437"}
Oct 30 03:12:14.343663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:14.343633    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 03:12:14.343857 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:14.343683    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 03:12:14.522126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:14.522079    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx" event={"ID":"919c258f-20ff-488a-a62e-9bc61dcd5319","Type":"ContainerStarted","Data":"f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27"}
Oct 30 03:12:14.721765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:14.719378    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:15.343439 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:15.343397    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 03:12:15.343624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:15.343470    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 03:12:16.343679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:16.343646    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 03:12:16.344089 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:16.343709    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 03:12:17.343094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:17.342670    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 03:12:17.343094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:17.342730    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 03:12:18.324450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.322020    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx" podStartSLOduration=6.321997236 podStartE2EDuration="6.321997236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:14.583741695 +0000 UTC m=+1872.507949880" watchObservedRunningTime="2026-10-30 03:12:18.321997236 +0000 UTC m=+1876.246205425"
Oct 30 03:12:18.329368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.327595    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"]
Oct 30 03:12:18.357450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.356450    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 03:12:18.357450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.356578    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 03:12:18.369481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.361077    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:18.385583 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.385448    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:12:18.469458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.466478    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"]
Oct 30 03:12:18.482449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.474912    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-734d43-5jtmc\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:18.482449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.474959    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9z4\" (UniqueName: \"kubernetes.io/projected/3d03f306-001c-49c7-85ea-abf44db0f896-kube-api-access-jq9z4\") pod \"maas-api-revoke-keys-e2e-api-b-734d43-5jtmc\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:18.482449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.475228    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-734d43-5jtmc\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:18.491568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.483176    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:12:18.579707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.576405    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-734d43-5jtmc\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:18.579707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.576462    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9z4\" (UniqueName: \"kubernetes.io/projected/3d03f306-001c-49c7-85ea-abf44db0f896-kube-api-access-jq9z4\") pod \"maas-api-revoke-keys-e2e-api-b-734d43-5jtmc\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:18.579707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.576668    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-734d43-5jtmc\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:18.579707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.577384    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-734d43-5jtmc\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:18.580492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.580417    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-734d43-5jtmc\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:18.599258 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.599228    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9z4\" (UniqueName: \"kubernetes.io/projected/3d03f306-001c-49c7-85ea-abf44db0f896-kube-api-access-jq9z4\") pod \"maas-api-revoke-keys-e2e-api-b-734d43-5jtmc\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:18.791452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:18.782716    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:19.199385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:19.199357    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"]
Oct 30 03:12:19.199597 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:12:19.199575    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d03f306_001c_49c7_85ea_abf44db0f896.slice/crio-789fb15a0615b6aa05f7341875869a011b551364b7ffd749490b16834a6db0f4 WatchSource:0}: Error finding container 789fb15a0615b6aa05f7341875869a011b551364b7ffd749490b16834a6db0f4: Status 404 returned error can't find the container with id 789fb15a0615b6aa05f7341875869a011b551364b7ffd749490b16834a6db0f4
Oct 30 03:12:19.348250 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:19.348221    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:19.348740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:19.348610    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:19.349488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:19.349466    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:12:19.636697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:19.636657    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc" event={"ID":"3d03f306-001c-49c7-85ea-abf44db0f896","Type":"ContainerStarted","Data":"789fb15a0615b6aa05f7341875869a011b551364b7ffd749490b16834a6db0f4"}
Oct 30 03:12:20.646055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:20.645964    4278 generic.go:356] "Generic (PLEG): container finished" podID="3d03f306-001c-49c7-85ea-abf44db0f896" containerID="d038ccb4948a057769a3ebf2f8fdd013222d0d26cca03f6d5ed9ac4ff5eb4edb" exitCode=0
Oct 30 03:12:20.646439 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:20.646047    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc" event={"ID":"3d03f306-001c-49c7-85ea-abf44db0f896","Type":"ContainerDied","Data":"d038ccb4948a057769a3ebf2f8fdd013222d0d26cca03f6d5ed9ac4ff5eb4edb"}
Oct 30 03:12:21.837749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:21.837723    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:21.945368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:21.945274    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-openshift-ca\") pod \"3d03f306-001c-49c7-85ea-abf44db0f896\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") "
Oct 30 03:12:21.945594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:21.945377    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d03f306-001c-49c7-85ea-abf44db0f896-kube-api-access-jq9z4\" (UniqueName: \"kubernetes.io/projected/3d03f306-001c-49c7-85ea-abf44db0f896-kube-api-access-jq9z4\") pod \"3d03f306-001c-49c7-85ea-abf44db0f896\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") "
Oct 30 03:12:21.945594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:21.945546    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-xks-ca\") pod \"3d03f306-001c-49c7-85ea-abf44db0f896\" (UID: \"3d03f306-001c-49c7-85ea-abf44db0f896\") "
Oct 30 03:12:21.945740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:21.945702    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-openshift-ca" pod "3d03f306-001c-49c7-85ea-abf44db0f896" (UID: "3d03f306-001c-49c7-85ea-abf44db0f896"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:21.946187 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:21.946162    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:21.947730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:21.947701    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-xks-ca" pod "3d03f306-001c-49c7-85ea-abf44db0f896" (UID: "3d03f306-001c-49c7-85ea-abf44db0f896"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:21.947827 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:21.947742    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d03f306-001c-49c7-85ea-abf44db0f896-kube-api-access-jq9z4" pod "3d03f306-001c-49c7-85ea-abf44db0f896" (UID: "3d03f306-001c-49c7-85ea-abf44db0f896"). InnerVolumeSpecName "kube-api-access-jq9z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:22.048498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:22.048469    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d03f306-001c-49c7-85ea-abf44db0f896-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:22.048498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:22.048499    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq9z4\" (UniqueName: \"kubernetes.io/projected/3d03f306-001c-49c7-85ea-abf44db0f896-kube-api-access-jq9z4\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:22.663382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:22.663352    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"
Oct 30 03:12:22.663382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:22.663360    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc" event={"ID":"3d03f306-001c-49c7-85ea-abf44db0f896","Type":"ContainerDied","Data":"789fb15a0615b6aa05f7341875869a011b551364b7ffd749490b16834a6db0f4"}
Oct 30 03:12:22.663687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:22.663406    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="789fb15a0615b6aa05f7341875869a011b551364b7ffd749490b16834a6db0f4"
Oct 30 03:12:22.718110 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:22.718076    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:22.732351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:22.732326    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:23.683838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:23.683807    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:12:24.191433 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:12:24.191387    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:12:41.982544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:41.980380    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:12:45.141519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:45.140538    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"]
Oct 30 03:12:45.150756 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:45.150698    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d30e9f-b8h8b"]
Oct 30 03:12:46.205659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:46.205625    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb0aee2-6472-46b9-9c0b-f13111e7f6f6" path="/var/lib/kubelet/pods/9fb0aee2-6472-46b9-9c0b-f13111e7f6f6/volumes"
Oct 30 03:12:51.921416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:51.921380    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"]
Oct 30 03:12:51.922302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:51.922239    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy" containerID="cri-o://97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889" gracePeriod=30
Oct 30 03:12:52.455962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.455920    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"]
Oct 30 03:12:52.456664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.456646    4278 cpu_manager.go:414] "containerMap: removing container" podUID="3d03f306-001c-49c7-85ea-abf44db0f896" containerName="revoke-keys"
Oct 30 03:12:52.456725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.456668    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d03f306-001c-49c7-85ea-abf44db0f896" containerName="revoke-keys"
Oct 30 03:12:52.456920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.456907    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d03f306-001c-49c7-85ea-abf44db0f896" containerName="revoke-keys"
Oct 30 03:12:52.462113 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.462090    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:52.480051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.480019    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"]
Oct 30 03:12:52.550989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.550948    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:12:52.551221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.551082    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:12:52.555717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.555681    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cea203-8l86b\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:52.555934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.555915    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cgtk\" (UniqueName: \"kubernetes.io/projected/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-kube-api-access-2cgtk\") pod \"maas-api-revoke-keys-e2e-inf-b-cea203-8l86b\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:52.556122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.556093    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cea203-8l86b\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:52.658365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.658319    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cea203-8l86b\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:52.658696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.658667    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cea203-8l86b\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:52.658872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.658851    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cgtk\" (UniqueName: \"kubernetes.io/projected/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-kube-api-access-2cgtk\") pod \"maas-api-revoke-keys-e2e-inf-b-cea203-8l86b\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:52.659203 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.659164    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cea203-8l86b\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:52.661300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.661272    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cea203-8l86b\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:52.679550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.679512    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cgtk\" (UniqueName: \"kubernetes.io/projected/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-kube-api-access-2cgtk\") pod \"maas-api-revoke-keys-e2e-inf-b-cea203-8l86b\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:52.780159 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:52.780122    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:53.097046 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:53.097006    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"]
Oct 30 03:12:53.097690 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:12:53.097664    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e7b4a9_bb9f_4ef6_92d3_43733f38e960.slice/crio-a20ee852d72758f6dae69201540d2ffedd54f55eb4f2c4f21505630bbc0c35fe WatchSource:0}: Error finding container a20ee852d72758f6dae69201540d2ffedd54f55eb4f2c4f21505630bbc0c35fe: Status 404 returned error can't find the container with id a20ee852d72758f6dae69201540d2ffedd54f55eb4f2c4f21505630bbc0c35fe
Oct 30 03:12:53.260477 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:53.260414    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b" event={"ID":"66e7b4a9-bb9f-4ef6-92d3-43733f38e960","Type":"ContainerStarted","Data":"a20ee852d72758f6dae69201540d2ffedd54f55eb4f2c4f21505630bbc0c35fe"}
Oct 30 03:12:54.201185 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:12:54.201142    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:12:54.272565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:54.272522    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b" event={"ID":"66e7b4a9-bb9f-4ef6-92d3-43733f38e960","Type":"ContainerStarted","Data":"ae6db0e6ed003f3319408ea89c081442da4c376dc82c5fed5d7dc1f12fdd8297"}
Oct 30 03:12:54.313668 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:54.313600    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b" podStartSLOduration=2.3135801 podStartE2EDuration="2.3135801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:54.313000927 +0000 UTC m=+1912.237209116" watchObservedRunningTime="2026-10-30 03:12:54.3135801 +0000 UTC m=+1912.237788288"
Oct 30 03:12:55.281995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:55.281962    4278 generic.go:356] "Generic (PLEG): container finished" podID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerID="ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb" exitCode=0
Oct 30 03:12:55.282617 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:55.282034    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" event={"ID":"dffbabd1-d7b2-4b61-aa48-58b0be40e9da","Type":"ContainerDied","Data":"ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb"}
Oct 30 03:12:55.282686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:55.282623    4278 scope.go:122] "RemoveContainer" containerID="ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb"
Oct 30 03:12:55.284146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:55.284122    4278 generic.go:356] "Generic (PLEG): container finished" podID="66e7b4a9-bb9f-4ef6-92d3-43733f38e960" containerID="ae6db0e6ed003f3319408ea89c081442da4c376dc82c5fed5d7dc1f12fdd8297" exitCode=0
Oct 30 03:12:55.284255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:55.284201    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b" event={"ID":"66e7b4a9-bb9f-4ef6-92d3-43733f38e960","Type":"ContainerDied","Data":"ae6db0e6ed003f3319408ea89c081442da4c376dc82c5fed5d7dc1f12fdd8297"}
Oct 30 03:12:55.284918 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:55.284890    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:55.284993 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:55.284980    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:55.285042 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:55.285030    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:56.306617 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.306579    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" event={"ID":"dffbabd1-d7b2-4b61-aa48-58b0be40e9da","Type":"ContainerStarted","Data":"baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93"}
Oct 30 03:12:56.531534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.531505    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:56.631178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.631081    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-kube-api-access-2cgtk\" (UniqueName: \"kubernetes.io/projected/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-kube-api-access-2cgtk\") pod \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") "
Oct 30 03:12:56.631358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.631257    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-xks-ca\") pod \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") "
Oct 30 03:12:56.631539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.631516    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-openshift-ca\") pod \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\" (UID: \"66e7b4a9-bb9f-4ef6-92d3-43733f38e960\") "
Oct 30 03:12:56.631887 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.631854    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-openshift-ca" pod "66e7b4a9-bb9f-4ef6-92d3-43733f38e960" (UID: "66e7b4a9-bb9f-4ef6-92d3-43733f38e960"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:56.632440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.632405    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:56.633527 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.633505    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-xks-ca" pod "66e7b4a9-bb9f-4ef6-92d3-43733f38e960" (UID: "66e7b4a9-bb9f-4ef6-92d3-43733f38e960"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:56.633693 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.633669    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-kube-api-access-2cgtk" pod "66e7b4a9-bb9f-4ef6-92d3-43733f38e960" (UID: "66e7b4a9-bb9f-4ef6-92d3-43733f38e960"). InnerVolumeSpecName "kube-api-access-2cgtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:56.734173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.734135    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cgtk\" (UniqueName: \"kubernetes.io/projected/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-kube-api-access-2cgtk\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:56.734173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.734165    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e7b4a9-bb9f-4ef6-92d3-43733f38e960-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:56.878451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.874551    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:12:56.878451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.875153    4278 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:12:56.878451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:56.875203    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 03:12:57.154042 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.154015    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:12:57.243408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.243311    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-envoy\") pod \"340bdf62-58b6-4dc0-9814-d3a91b88e949\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") "
Oct 30 03:12:57.243408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.243385    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-podinfo\") pod \"340bdf62-58b6-4dc0-9814-d3a91b88e949\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") "
Oct 30 03:12:57.243727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.243567    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-socket\") pod \"340bdf62-58b6-4dc0-9814-d3a91b88e949\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") "
Oct 30 03:12:57.243727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.243600    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-certs\") pod \"340bdf62-58b6-4dc0-9814-d3a91b88e949\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") "
Oct 30 03:12:57.243727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.243660    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/340bdf62-58b6-4dc0-9814-d3a91b88e949-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340bdf62-58b6-4dc0-9814-d3a91b88e949-istiod-ca-cert\") pod \"340bdf62-58b6-4dc0-9814-d3a91b88e949\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") "
Oct 30 03:12:57.243727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.243721    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-token\" (UniqueName: \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-token\") pod \"340bdf62-58b6-4dc0-9814-d3a91b88e949\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") "
Oct 30 03:12:57.243944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.243796    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-data\") pod \"340bdf62-58b6-4dc0-9814-d3a91b88e949\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") "
Oct 30 03:12:57.243944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.243824    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-credential-socket\") pod \"340bdf62-58b6-4dc0-9814-d3a91b88e949\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") "
Oct 30 03:12:57.243944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.243892    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-kube-api-access-8v5vv\" (UniqueName: \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-kube-api-access-8v5vv\") pod \"340bdf62-58b6-4dc0-9814-d3a91b88e949\" (UID: \"340bdf62-58b6-4dc0-9814-d3a91b88e949\") "
Oct 30 03:12:57.244097 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.244021    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-socket" pod "340bdf62-58b6-4dc0-9814-d3a91b88e949" (UID: "340bdf62-58b6-4dc0-9814-d3a91b88e949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:57.244158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.244119    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340bdf62-58b6-4dc0-9814-d3a91b88e949-istiod-ca-cert" pod "340bdf62-58b6-4dc0-9814-d3a91b88e949" (UID: "340bdf62-58b6-4dc0-9814-d3a91b88e949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:57.244281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.244257    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-data" pod "340bdf62-58b6-4dc0-9814-d3a91b88e949" (UID: "340bdf62-58b6-4dc0-9814-d3a91b88e949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:57.244550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.244524    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-credential-socket" pod "340bdf62-58b6-4dc0-9814-d3a91b88e949" (UID: "340bdf62-58b6-4dc0-9814-d3a91b88e949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:57.244916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.244892    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.245012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.244924    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340bdf62-58b6-4dc0-9814-d3a91b88e949-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.245012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.244938    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.245012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.244952    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.245394 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.245367    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-certs" pod "340bdf62-58b6-4dc0-9814-d3a91b88e949" (UID: "340bdf62-58b6-4dc0-9814-d3a91b88e949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:57.247110 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.247087    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-token" pod "340bdf62-58b6-4dc0-9814-d3a91b88e949" (UID: "340bdf62-58b6-4dc0-9814-d3a91b88e949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:57.247220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.247085    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-envoy" pod "340bdf62-58b6-4dc0-9814-d3a91b88e949" (UID: "340bdf62-58b6-4dc0-9814-d3a91b88e949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:57.247302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.247235    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-kube-api-access-8v5vv" pod "340bdf62-58b6-4dc0-9814-d3a91b88e949" (UID: "340bdf62-58b6-4dc0-9814-d3a91b88e949"). InnerVolumeSpecName "kube-api-access-8v5vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:57.247697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.247676    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-podinfo" pod "340bdf62-58b6-4dc0-9814-d3a91b88e949" (UID: "340bdf62-58b6-4dc0-9814-d3a91b88e949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:12:57.316236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.316197    4278 generic.go:356] "Generic (PLEG): container finished" podID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerID="97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889" exitCode=0
Oct 30 03:12:57.316728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.316259    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"
Oct 30 03:12:57.316728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.316280    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" event={"ID":"340bdf62-58b6-4dc0-9814-d3a91b88e949","Type":"ContainerDied","Data":"97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889"}
Oct 30 03:12:57.316728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.316339    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4" event={"ID":"340bdf62-58b6-4dc0-9814-d3a91b88e949","Type":"ContainerDied","Data":"dee0fcc0852fc692ce6fff9950c840c95d9ce1cbed3304b94d703beebdbff031"}
Oct 30 03:12:57.316728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.316373    4278 scope.go:122] "RemoveContainer" containerID="97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889"
Oct 30 03:12:57.318026 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.317993    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b" event={"ID":"66e7b4a9-bb9f-4ef6-92d3-43733f38e960","Type":"ContainerDied","Data":"a20ee852d72758f6dae69201540d2ffedd54f55eb4f2c4f21505630bbc0c35fe"}
Oct 30 03:12:57.318026 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.318007    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"
Oct 30 03:12:57.318181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.318041    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a20ee852d72758f6dae69201540d2ffedd54f55eb4f2c4f21505630bbc0c35fe"
Oct 30 03:12:57.330359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.330334    4278 scope.go:122] "RemoveContainer" containerID="296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af"
Oct 30 03:12:57.342092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.342069    4278 scope.go:122] "RemoveContainer" containerID="97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889"
Oct 30 03:12:57.342436 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:12:57.342399    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889\": container with ID starting with 97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889 not found: ID does not exist" containerID="97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889"
Oct 30 03:12:57.342543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.342512    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889"} err="failed to get container status \"97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889\": rpc error: code = NotFound desc = could not find container \"97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889\": container with ID starting with 97932a859e7b0991435a0596ce06e2110c39bfce4b7798339b7056c3d2fae889 not found: ID does not exist"
Oct 30 03:12:57.342543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.342533    4278 scope.go:122] "RemoveContainer" containerID="296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af"
Oct 30 03:12:57.342808 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:12:57.342792    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af\": container with ID starting with 296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af not found: ID does not exist" containerID="296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af"
Oct 30 03:12:57.342872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.342813    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af"} err="failed to get container status \"296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af\": rpc error: code = NotFound desc = could not find container \"296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af\": container with ID starting with 296c5d97822ac96adff1668e602f67e388583f9f1b4044c42cfc43802c97c9af not found: ID does not exist"
Oct 30 03:12:57.347800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.347768    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v5vv\" (UniqueName: \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-kube-api-access-8v5vv\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.347915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.347803    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.347915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.347821    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.347915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.347835    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340bdf62-58b6-4dc0-9814-d3a91b88e949-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.347915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.347849    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340bdf62-58b6-4dc0-9814-d3a91b88e949-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.410813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.410769    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"]
Oct 30 03:12:57.422892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.422858    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d30e9f-openshift-default-776cc5cb65-lxzj4"]
Oct 30 03:12:57.877524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.876645    4278 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:12:57.877524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:57.876722    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 03:12:58.202571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:58.202491    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" path="/var/lib/kubelet/pods/340bdf62-58b6-4dc0-9814-d3a91b88e949/volumes"
Oct 30 03:12:58.883667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:58.883623    4278 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:12:58.884187 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:58.883697    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 03:12:59.883859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:59.883820    4278 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:12:59.884494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:12:59.884443    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 03:13:00.883464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:00.872695    4278 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:13:00.883464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:00.872760    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 03:13:01.878848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:01.878813    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:13:01.879386 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:01.879362    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:13:01.880145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:01.880126    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:13:03.380954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:03.380918    4278 scope.go:122] "RemoveContainer" containerID="4545cd9e4f317415951a14725995ca885a9f76569251b374b572b2f1e6fcd8a0"
Oct 30 03:13:03.392239 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:03.392215    4278 scope.go:122] "RemoveContainer" containerID="52a498cb726ee1776cb4bebb8bcf7c67b7a392838111fb6bc96afa9854327cb2"
Oct 30 03:13:11.634498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.634454    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"]
Oct 30 03:13:11.635650 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.635626    4278 cpu_manager.go:414] "containerMap: removing container" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy"
Oct 30 03:13:11.635724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.635654    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy"
Oct 30 03:13:11.635724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.635692    4278 cpu_manager.go:414] "containerMap: removing container" podUID="66e7b4a9-bb9f-4ef6-92d3-43733f38e960" containerName="revoke-keys"
Oct 30 03:13:11.635724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.635700    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e7b4a9-bb9f-4ef6-92d3-43733f38e960" containerName="revoke-keys"
Oct 30 03:13:11.635817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.635729    4278 cpu_manager.go:414] "containerMap: removing container" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy"
Oct 30 03:13:11.635817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.635737    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy"
Oct 30 03:13:11.635915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.635901    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy"
Oct 30 03:13:11.635958 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.635923    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e7b4a9-bb9f-4ef6-92d3-43733f38e960" containerName="revoke-keys"
Oct 30 03:13:11.635958 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.635940    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="340bdf62-58b6-4dc0-9814-d3a91b88e949" containerName="istio-proxy"
Oct 30 03:13:11.640953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.640931    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.674310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.674270    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"]
Oct 30 03:13:11.685784 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.685751    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4f415d\"/\"test-model-4f415d-kserve-self-signed-certs\""
Oct 30 03:13:11.685987 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.685802    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4f415d\"/\"openshift-service-ca.crt\""
Oct 30 03:13:11.685987 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.685847    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4f415d\"/\"default-dockercfg-mztcm\""
Oct 30 03:13:11.685987 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.685926    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4f415d\"/\"kube-root-ca.crt\""
Oct 30 03:13:11.707076 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.707040    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-tmp-dir\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.707405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.707372    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-dshm\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.707624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.707495    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-model-cache\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.707685 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.707650    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a39628b-20dd-4169-a2f5-672963350cfa-tls-certs\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.707828 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.707807    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrs9\" (UniqueName: \"kubernetes.io/projected/2a39628b-20dd-4169-a2f5-672963350cfa-kube-api-access-bmrs9\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.707894 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.707856    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-home\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.809359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.809310    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-dshm\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.809359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.809366    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-model-cache\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.809692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.809475    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a39628b-20dd-4169-a2f5-672963350cfa-tls-certs\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.809692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.809621    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrs9\" (UniqueName: \"kubernetes.io/projected/2a39628b-20dd-4169-a2f5-672963350cfa-kube-api-access-bmrs9\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.809692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.809664    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-home\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.809867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.809824    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-tmp-dir\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.809924 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.809865    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-model-cache\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.810172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.810149    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-tmp-dir\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.810259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.810222    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-home\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.812542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.812508    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a39628b-20dd-4169-a2f5-672963350cfa-tls-certs\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.812542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.812531    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-dshm\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.830556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.830521    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrs9\" (UniqueName: \"kubernetes.io/projected/2a39628b-20dd-4169-a2f5-672963350cfa-kube-api-access-bmrs9\") pod \"test-model-4f415d-kserve-64fbb5585f-gqn6f\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") " pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:11.955146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:11.954768    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:12.401846 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:12.401811    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"]
Oct 30 03:13:12.402757 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:13:12.402713    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a39628b_20dd_4169_a2f5_672963350cfa.slice/crio-dbbc4c07228471aa9f243f7f5013b3c1fe24383e8ed520411d7912ca961513e8 WatchSource:0}: Error finding container dbbc4c07228471aa9f243f7f5013b3c1fe24383e8ed520411d7912ca961513e8: Status 404 returned error can't find the container with id dbbc4c07228471aa9f243f7f5013b3c1fe24383e8ed520411d7912ca961513e8
Oct 30 03:13:12.486345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:12.486302    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f" event={"ID":"2a39628b-20dd-4169-a2f5-672963350cfa","Type":"ContainerStarted","Data":"4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73"}
Oct 30 03:13:12.486520 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:12.486360    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f" event={"ID":"2a39628b-20dd-4169-a2f5-672963350cfa","Type":"ContainerStarted","Data":"dbbc4c07228471aa9f243f7f5013b3c1fe24383e8ed520411d7912ca961513e8"}
Oct 30 03:13:12.534415 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:12.534358    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f" podStartSLOduration=1.5343403009999999 podStartE2EDuration="1.534340301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:12.53014971 +0000 UTC m=+1930.454357899" watchObservedRunningTime="2026-10-30 03:13:12.534340301 +0000 UTC m=+1930.458548589"
Oct 30 03:13:12.956362 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:12.956321    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:21.956629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:21.956541    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:21.969661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:21.969637    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:22.616852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:22.616820    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:13:24.197662 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:13:24.197626    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:13:32.071067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:32.071029    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"]
Oct 30 03:13:32.086632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:32.086596    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cea203-8l86b"]
Oct 30 03:13:32.206326 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:32.206291    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e7b4a9-bb9f-4ef6-92d3-43733f38e960" path="/var/lib/kubelet/pods/66e7b4a9-bb9f-4ef6-92d3-43733f38e960/volumes"
Oct 30 03:13:36.397661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.397622    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"]
Oct 30 03:13:36.403415 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.403387    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:36.421827 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.421796    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"]
Oct 30 03:13:36.428404 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.428371    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:13:36.428596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.428501    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:13:36.544044 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.544011    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e553ae-858rr\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:36.544221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.544137    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e553ae-858rr\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:36.544268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.544238    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlxlv\" (UniqueName: \"kubernetes.io/projected/7c928965-d2d6-461f-953a-0ea65490b797-kube-api-access-jlxlv\") pod \"maas-api-revoke-sub-b3e553ae-858rr\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:36.645315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.645280    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e553ae-858rr\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:36.645551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.645366    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e553ae-858rr\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:36.645551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.645443    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlxlv\" (UniqueName: \"kubernetes.io/projected/7c928965-d2d6-461f-953a-0ea65490b797-kube-api-access-jlxlv\") pod \"maas-api-revoke-sub-b3e553ae-858rr\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:36.646083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.646055    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e553ae-858rr\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:36.647752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.647696    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e553ae-858rr\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:36.665545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.665511    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlxlv\" (UniqueName: \"kubernetes.io/projected/7c928965-d2d6-461f-953a-0ea65490b797-kube-api-access-jlxlv\") pod \"maas-api-revoke-sub-b3e553ae-858rr\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:36.731782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:36.731750    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:37.033102 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:37.033072    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"]
Oct 30 03:13:37.034253 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:13:37.034226    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c928965_d2d6_461f_953a_0ea65490b797.slice/crio-2449e6dde5b6600860e772fcbbb4fb8f324b8879cec4acb89b13f3410494dc85 WatchSource:0}: Error finding container 2449e6dde5b6600860e772fcbbb4fb8f324b8879cec4acb89b13f3410494dc85: Status 404 returned error can't find the container with id 2449e6dde5b6600860e772fcbbb4fb8f324b8879cec4acb89b13f3410494dc85
Oct 30 03:13:37.828017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:37.827978    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr" event={"ID":"7c928965-d2d6-461f-953a-0ea65490b797","Type":"ContainerStarted","Data":"f40f720dc848be0abf5e70cca496bfc914962f20fb379a839b7a18ad96b6c7fa"}
Oct 30 03:13:37.828553 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:37.828040    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr" event={"ID":"7c928965-d2d6-461f-953a-0ea65490b797","Type":"ContainerStarted","Data":"2449e6dde5b6600860e772fcbbb4fb8f324b8879cec4acb89b13f3410494dc85"}
Oct 30 03:13:37.868283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:37.868225    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr" podStartSLOduration=1.868207631 podStartE2EDuration="1.868207631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:37.867071094 +0000 UTC m=+1955.791279283" watchObservedRunningTime="2026-10-30 03:13:37.868207631 +0000 UTC m=+1955.792415819"
Oct 30 03:13:38.837563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:38.837526    4278 generic.go:356] "Generic (PLEG): container finished" podID="7c928965-d2d6-461f-953a-0ea65490b797" containerID="f40f720dc848be0abf5e70cca496bfc914962f20fb379a839b7a18ad96b6c7fa" exitCode=0
Oct 30 03:13:38.838024 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:38.837583    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr" event={"ID":"7c928965-d2d6-461f-953a-0ea65490b797","Type":"ContainerDied","Data":"f40f720dc848be0abf5e70cca496bfc914962f20fb379a839b7a18ad96b6c7fa"}
Oct 30 03:13:40.016470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.016444    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:40.104278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.104179    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-xks-ca\") pod \"7c928965-d2d6-461f-953a-0ea65490b797\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") "
Oct 30 03:13:40.104278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.104222    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-openshift-ca\") pod \"7c928965-d2d6-461f-953a-0ea65490b797\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") "
Oct 30 03:13:40.104544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.104406    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c928965-d2d6-461f-953a-0ea65490b797-kube-api-access-jlxlv\" (UniqueName: \"kubernetes.io/projected/7c928965-d2d6-461f-953a-0ea65490b797-kube-api-access-jlxlv\") pod \"7c928965-d2d6-461f-953a-0ea65490b797\" (UID: \"7c928965-d2d6-461f-953a-0ea65490b797\") "
Oct 30 03:13:40.104904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.104876    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-openshift-ca" pod "7c928965-d2d6-461f-953a-0ea65490b797" (UID: "7c928965-d2d6-461f-953a-0ea65490b797"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:40.105320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.105296    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:13:40.106832 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.106807    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c928965-d2d6-461f-953a-0ea65490b797-kube-api-access-jlxlv" pod "7c928965-d2d6-461f-953a-0ea65490b797" (UID: "7c928965-d2d6-461f-953a-0ea65490b797"). InnerVolumeSpecName "kube-api-access-jlxlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:40.107106 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.107087    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-xks-ca" pod "7c928965-d2d6-461f-953a-0ea65490b797" (UID: "7c928965-d2d6-461f-953a-0ea65490b797"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:40.206974 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.206936    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlxlv\" (UniqueName: \"kubernetes.io/projected/7c928965-d2d6-461f-953a-0ea65490b797-kube-api-access-jlxlv\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:13:40.206974 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.206970    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c928965-d2d6-461f-953a-0ea65490b797-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:13:40.867964 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.867922    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr" event={"ID":"7c928965-d2d6-461f-953a-0ea65490b797","Type":"ContainerDied","Data":"2449e6dde5b6600860e772fcbbb4fb8f324b8879cec4acb89b13f3410494dc85"}
Oct 30 03:13:40.868347 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.868330    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2449e6dde5b6600860e772fcbbb4fb8f324b8879cec4acb89b13f3410494dc85"
Oct 30 03:13:40.868500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:40.868276    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"
Oct 30 03:13:46.818480 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.818435    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"]
Oct 30 03:13:46.819251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.819232    4278 cpu_manager.go:414] "containerMap: removing container" podUID="7c928965-d2d6-461f-953a-0ea65490b797" containerName="revoke-keys"
Oct 30 03:13:46.819309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.819253    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c928965-d2d6-461f-953a-0ea65490b797" containerName="revoke-keys"
Oct 30 03:13:46.819408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.819399    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c928965-d2d6-461f-953a-0ea65490b797" containerName="revoke-keys"
Oct 30 03:13:46.823653 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.823635    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:46.842619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.842584    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"]
Oct 30 03:13:46.849253 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.849215    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:13:46.849517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.849490    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:13:46.946437 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.946369    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88627bdf-r674s\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:46.946744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.946724    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88627bdf-r674s\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:46.946839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:46.946758    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5zm\" (UniqueName: \"kubernetes.io/projected/bb3e963a-e132-444b-8ff5-240ee0fa94b6-kube-api-access-mg5zm\") pod \"maas-api-revoke-sub-88627bdf-r674s\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:47.048001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:47.047967    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88627bdf-r674s\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:47.048192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:47.048010    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5zm\" (UniqueName: \"kubernetes.io/projected/bb3e963a-e132-444b-8ff5-240ee0fa94b6-kube-api-access-mg5zm\") pod \"maas-api-revoke-sub-88627bdf-r674s\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:47.048192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:47.048088    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88627bdf-r674s\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:47.048757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:47.048729    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88627bdf-r674s\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:47.050580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:47.050556    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88627bdf-r674s\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:47.074329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:47.074258    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5zm\" (UniqueName: \"kubernetes.io/projected/bb3e963a-e132-444b-8ff5-240ee0fa94b6-kube-api-access-mg5zm\") pod \"maas-api-revoke-sub-88627bdf-r674s\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:47.139632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:47.139594    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:47.453911 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:13:47.453872    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3e963a_e132_444b_8ff5_240ee0fa94b6.slice/crio-44b32d87f496a4f0a222798458058472004bb77f9e39ca21f68fcfe10c82d813 WatchSource:0}: Error finding container 44b32d87f496a4f0a222798458058472004bb77f9e39ca21f68fcfe10c82d813: Status 404 returned error can't find the container with id 44b32d87f496a4f0a222798458058472004bb77f9e39ca21f68fcfe10c82d813
Oct 30 03:13:47.455147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:47.455114    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"]
Oct 30 03:13:47.925002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:47.924962    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s" event={"ID":"bb3e963a-e132-444b-8ff5-240ee0fa94b6","Type":"ContainerStarted","Data":"44b32d87f496a4f0a222798458058472004bb77f9e39ca21f68fcfe10c82d813"}
Oct 30 03:13:48.935549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:48.935450    4278 generic.go:356] "Generic (PLEG): container finished" podID="bb3e963a-e132-444b-8ff5-240ee0fa94b6" containerID="2adc41119f83177f95eaed385cbbc44a7551bab729dd68d7231c57e7c40bf78f" exitCode=0
Oct 30 03:13:48.935549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:48.935533    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s" event={"ID":"bb3e963a-e132-444b-8ff5-240ee0fa94b6","Type":"ContainerDied","Data":"2adc41119f83177f95eaed385cbbc44a7551bab729dd68d7231c57e7c40bf78f"}
Oct 30 03:13:50.165804 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.165771    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:50.317013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.316974    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-openshift-ca\") pod \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") "
Oct 30 03:13:50.317013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.317023    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-xks-ca\") pod \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") "
Oct 30 03:13:50.317298 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.317288    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3e963a-e132-444b-8ff5-240ee0fa94b6-kube-api-access-mg5zm\" (UniqueName: \"kubernetes.io/projected/bb3e963a-e132-444b-8ff5-240ee0fa94b6-kube-api-access-mg5zm\") pod \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\" (UID: \"bb3e963a-e132-444b-8ff5-240ee0fa94b6\") "
Oct 30 03:13:50.317437 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.317397    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-openshift-ca" pod "bb3e963a-e132-444b-8ff5-240ee0fa94b6" (UID: "bb3e963a-e132-444b-8ff5-240ee0fa94b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:50.317966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.317945    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:13:50.319224 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.319194    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-xks-ca" pod "bb3e963a-e132-444b-8ff5-240ee0fa94b6" (UID: "bb3e963a-e132-444b-8ff5-240ee0fa94b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:50.319385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.319367    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3e963a-e132-444b-8ff5-240ee0fa94b6-kube-api-access-mg5zm" pod "bb3e963a-e132-444b-8ff5-240ee0fa94b6" (UID: "bb3e963a-e132-444b-8ff5-240ee0fa94b6"). InnerVolumeSpecName "kube-api-access-mg5zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:50.420128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.420089    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3e963a-e132-444b-8ff5-240ee0fa94b6-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:13:50.420128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.420130    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5zm\" (UniqueName: \"kubernetes.io/projected/bb3e963a-e132-444b-8ff5-240ee0fa94b6-kube-api-access-mg5zm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:13:50.953277 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.953246    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"
Oct 30 03:13:50.953533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.953277    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s" event={"ID":"bb3e963a-e132-444b-8ff5-240ee0fa94b6","Type":"ContainerDied","Data":"44b32d87f496a4f0a222798458058472004bb77f9e39ca21f68fcfe10c82d813"}
Oct 30 03:13:50.953533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:50.953403    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44b32d87f496a4f0a222798458058472004bb77f9e39ca21f68fcfe10c82d813"
Oct 30 03:13:52.494341 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:52.494288    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"]
Oct 30 03:13:52.507375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:52.507338    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-734d43-5jtmc"]
Oct 30 03:13:54.191913 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:13:54.191868    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:13:54.201536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:13:54.201500    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d03f306-001c-49c7-85ea-abf44db0f896" path="/var/lib/kubelet/pods/3d03f306-001c-49c7-85ea-abf44db0f896/volumes"
Oct 30 03:14:03.447090 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:03.447054    4278 scope.go:122] "RemoveContainer" containerID="a57a818e1b6fd585271b4bf1865235a127aeeb8ad82dfdc1c25b94b56a8e2475"
Oct 30 03:14:03.460451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:03.460372    4278 scope.go:122] "RemoveContainer" containerID="c2cc6eebc68b51ad66dd283e18e6ce117453c58fba45d8c04d7a7e64e5978c68"
Oct 30 03:14:03.471402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:03.471376    4278 scope.go:122] "RemoveContainer" containerID="67c1559199191ed9f37a3b8d41faa9f358e7e071bc9c6c6f3aa89a418dd72568"
Oct 30 03:14:19.864049 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:19.864007    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:14:23.036700 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:23.036638    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4" podUID="f66096de-fc52-463a-a740-09bc1a1692d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.144:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:14:23.190723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:23.190650    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:14:23.804657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:23.804611    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"]
Oct 30 03:14:24.204914 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:14:24.204821    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:14:33.821110 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.821068    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"]
Oct 30 03:14:33.822153 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.822129    4278 cpu_manager.go:414] "containerMap: removing container" podUID="bb3e963a-e132-444b-8ff5-240ee0fa94b6" containerName="revoke-keys"
Oct 30 03:14:33.822259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.822155    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3e963a-e132-444b-8ff5-240ee0fa94b6" containerName="revoke-keys"
Oct 30 03:14:33.822359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.822347    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3e963a-e132-444b-8ff5-240ee0fa94b6" containerName="revoke-keys"
Oct 30 03:14:33.825898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.825877    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:33.847294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.847262    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"]
Oct 30 03:14:33.854334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.854296    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-360818-openshift-default-dockercfg-clvks\""
Oct 30 03:14:33.921330 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.921295    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:33.921534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.921347    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:33.921534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.921374    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60341dc-c799-4814-8a04-d37ac0d30993-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:33.921534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.921520    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:33.921658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.921544    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:33.921694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.921665    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:33.921729 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.921709    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:33.921765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.921758    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60341dc-c799-4814-8a04-d37ac0d30993-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:33.921944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:33.921922    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6nbq\" (UniqueName: \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-kube-api-access-p6nbq\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.023782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.023742    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6nbq\" (UniqueName: \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-kube-api-access-p6nbq\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.023956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.023871    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.023956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.023899    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.023956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.023919    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60341dc-c799-4814-8a04-d37ac0d30993-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.024129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.023976    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.024129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.023999    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.024129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.024062    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.024129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.024099    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.024334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.024165    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60341dc-c799-4814-8a04-d37ac0d30993-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.024394 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.024341    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.024467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.024391    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.024523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.024502    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.024580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.024556    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.024976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.024959    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60341dc-c799-4814-8a04-d37ac0d30993-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.026466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.026404    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.026710 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.026689    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60341dc-c799-4814-8a04-d37ac0d30993-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.043962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.043934    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.044098 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.044041    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6nbq\" (UniqueName: \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-kube-api-access-p6nbq\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.140766 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.140658    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:34.469751 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.469713    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"]
Oct 30 03:14:34.471035 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:14:34.471004    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60341dc_c799_4814_8a04_d37ac0d30993.slice/crio-326afe2dfb13e5d67163b917f7198623fd7f6c49f50c7a6a04e148843e35730d WatchSource:0}: Error finding container 326afe2dfb13e5d67163b917f7198623fd7f6c49f50c7a6a04e148843e35730d: Status 404 returned error can't find the container with id 326afe2dfb13e5d67163b917f7198623fd7f6c49f50c7a6a04e148843e35730d
Oct 30 03:14:34.474796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.474756    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:14:34.474898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.474831    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:14:34.474898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.474868    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:14:34.542467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:34.542433    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" event={"ID":"d60341dc-c799-4814-8a04-d37ac0d30993","Type":"ContainerStarted","Data":"326afe2dfb13e5d67163b917f7198623fd7f6c49f50c7a6a04e148843e35730d"}
Oct 30 03:14:35.555138 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:35.555096    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" event={"ID":"d60341dc-c799-4814-8a04-d37ac0d30993","Type":"ContainerStarted","Data":"16cf572a8490a2d705d1598ab9fe2529a85b2b304d8c37997f738b80e3c65daf"}
Oct 30 03:14:36.141185 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:36.141147    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:36.143206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:36.143183    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 03:14:36.143270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:36.143238    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 03:14:37.141667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:37.141630    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 03:14:37.142173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:37.141695    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 03:14:38.141280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:38.141245    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 03:14:38.141482 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:38.141305    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 03:14:38.830187 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:38.830141    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4" podUID="f66096de-fc52-463a-a740-09bc1a1692d0" containerName="main" containerID="cri-o://3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5" gracePeriod=45
Oct 30 03:14:38.911317 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:38.911224    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" podStartSLOduration=5.911201938 podStartE2EDuration="5.911201938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:35.614787783 +0000 UTC m=+2013.538995971" watchObservedRunningTime="2026-10-30 03:14:38.911201938 +0000 UTC m=+2016.835410126"
Oct 30 03:14:38.938557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:38.924147    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"]
Oct 30 03:14:38.976182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:38.975865    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4" podUID="f66096de-fc52-463a-a740-09bc1a1692d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.144:8000/ready\": dial tcp 10.134.0.144:8000: connect: connection refused"
Oct 30 03:14:39.149513 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.146037    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 03:14:39.149513 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.146105    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 03:14:39.508806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.508775    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:14:39.566452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.558981    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66096de-fc52-463a-a740-09bc1a1692d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/f66096de-fc52-463a-a740-09bc1a1692d0-tls-certs\") pod \"f66096de-fc52-463a-a740-09bc1a1692d0\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") "
Oct 30 03:14:39.566452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.559084    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-home\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-home\") pod \"f66096de-fc52-463a-a740-09bc1a1692d0\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") "
Oct 30 03:14:39.566452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.559131    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-tmp-dir\") pod \"f66096de-fc52-463a-a740-09bc1a1692d0\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") "
Oct 30 03:14:39.566452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.559209    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66096de-fc52-463a-a740-09bc1a1692d0-kube-api-access-fq6ht\" (UniqueName: \"kubernetes.io/projected/f66096de-fc52-463a-a740-09bc1a1692d0-kube-api-access-fq6ht\") pod \"f66096de-fc52-463a-a740-09bc1a1692d0\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") "
Oct 30 03:14:39.566452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.559309    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-dshm\") pod \"f66096de-fc52-463a-a740-09bc1a1692d0\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") "
Oct 30 03:14:39.566452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.559348    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-model-cache\") pod \"f66096de-fc52-463a-a740-09bc1a1692d0\" (UID: \"f66096de-fc52-463a-a740-09bc1a1692d0\") "
Oct 30 03:14:39.566452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.562737    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-tmp-dir" pod "f66096de-fc52-463a-a740-09bc1a1692d0" (UID: "f66096de-fc52-463a-a740-09bc1a1692d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:39.576744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.571541    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-home" pod "f66096de-fc52-463a-a740-09bc1a1692d0" (UID: "f66096de-fc52-463a-a740-09bc1a1692d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:39.576744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.574087    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:39.576744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.574118    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:39.576744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.574200    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-model-cache" pod "f66096de-fc52-463a-a740-09bc1a1692d0" (UID: "f66096de-fc52-463a-a740-09bc1a1692d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:39.605450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.603556    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66096de-fc52-463a-a740-09bc1a1692d0-tls-certs" pod "f66096de-fc52-463a-a740-09bc1a1692d0" (UID: "f66096de-fc52-463a-a740-09bc1a1692d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:14:39.609466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.609150    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66096de-fc52-463a-a740-09bc1a1692d0-kube-api-access-fq6ht" pod "f66096de-fc52-463a-a740-09bc1a1692d0" (UID: "f66096de-fc52-463a-a740-09bc1a1692d0"). InnerVolumeSpecName "kube-api-access-fq6ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:39.617103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.611174    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-dshm" pod "f66096de-fc52-463a-a740-09bc1a1692d0" (UID: "f66096de-fc52-463a-a740-09bc1a1692d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:39.617103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.616522    4278 generic.go:356] "Generic (PLEG): container finished" podID="f66096de-fc52-463a-a740-09bc1a1692d0" containerID="3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5" exitCode=0
Oct 30 03:14:39.617103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.616696    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4" event={"ID":"f66096de-fc52-463a-a740-09bc1a1692d0","Type":"ContainerDied","Data":"3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5"}
Oct 30 03:14:39.617103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.616723    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4" event={"ID":"f66096de-fc52-463a-a740-09bc1a1692d0","Type":"ContainerDied","Data":"2b2c531ecbc0a34e6368e2828c3a443df4379d5607ab634fe2bee9a5c3204c38"}
Oct 30 03:14:39.617103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.616739    4278 scope.go:122] "RemoveContainer" containerID="3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5"
Oct 30 03:14:39.617103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.616876    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"
Oct 30 03:14:39.634998 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.634977    4278 scope.go:122] "RemoveContainer" containerID="3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5"
Oct 30 03:14:39.635328 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:14:39.635306    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5\": container with ID starting with 3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5 not found: ID does not exist" containerID="3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5"
Oct 30 03:14:39.635418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.635343    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5"} err="failed to get container status \"3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5\": rpc error: code = NotFound desc = could not find container \"3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5\": container with ID starting with 3b1ee19a76ed031d48163614994f4f0bb5f585e0a0b64bca89861eca2829b0c5 not found: ID does not exist"
Oct 30 03:14:39.689533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.689476    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6ht\" (UniqueName: \"kubernetes.io/projected/f66096de-fc52-463a-a740-09bc1a1692d0-kube-api-access-fq6ht\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:39.689533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.689531    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:39.689796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.689547    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66096de-fc52-463a-a740-09bc1a1692d0-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:39.689796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.689560    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66096de-fc52-463a-a740-09bc1a1692d0-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:39.694759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.694645    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"]
Oct 30 03:14:39.700521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:39.700493    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/auth-test-model-9245d1-kserve-7c5b466b8f-qfrt4"]
Oct 30 03:14:40.142956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:40.142913    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 03:14:40.143469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:40.142986    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 03:14:40.202077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:40.202039    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66096de-fc52-463a-a740-09bc1a1692d0" path="/var/lib/kubelet/pods/f66096de-fc52-463a-a740-09bc1a1692d0/volumes"
Oct 30 03:14:41.141565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:41.141522    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 03:14:41.141768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:41.141593    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 03:14:42.147204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:42.147170    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:42.147680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:42.147504    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:42.148472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:42.148448    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:14:51.029509 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:51.029398    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"]
Oct 30 03:14:51.042348 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:51.042307    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6058e71-pxpz5"]
Oct 30 03:14:52.203187 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:52.203140    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f884cf6-bf41-45f7-837b-63b591cd2af2" path="/var/lib/kubelet/pods/9f884cf6-bf41-45f7-837b-63b591cd2af2/volumes"
Oct 30 03:14:54.036662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:54.036616    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt" podUID="80e943c9-b7ad-4d36-866d-ae0fe597fca1" containerName="main" containerID="cri-o://53c6e3527fff38108492f9004b87e4d12081da31ff5211d02d3e1fac390e8829" gracePeriod=45
Oct 30 03:14:54.337773 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:14:54.337681    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:14:54.351001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:54.350931    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt" podUID="80e943c9-b7ad-4d36-866d-ae0fe597fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.145:8000/ready\": dial tcp 10.134.0.145:8000: connect: connection refused"
Oct 30 03:14:54.833013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:54.832975    4278 generic.go:356] "Generic (PLEG): container finished" podID="80e943c9-b7ad-4d36-866d-ae0fe597fca1" containerID="53c6e3527fff38108492f9004b87e4d12081da31ff5211d02d3e1fac390e8829" exitCode=0
Oct 30 03:14:54.833164 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:54.833027    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt" event={"ID":"80e943c9-b7ad-4d36-866d-ae0fe597fca1","Type":"ContainerDied","Data":"53c6e3527fff38108492f9004b87e4d12081da31ff5211d02d3e1fac390e8829"}
Oct 30 03:14:54.950773 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:54.950743    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:14:55.041035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.040824    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tls-certs\" (UniqueName: \"kubernetes.io/secret/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tls-certs\") pod \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") "
Oct 30 03:14:55.041035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.040909    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e943c9-b7ad-4d36-866d-ae0fe597fca1-kube-api-access-7dwtf\" (UniqueName: \"kubernetes.io/projected/80e943c9-b7ad-4d36-866d-ae0fe597fca1-kube-api-access-7dwtf\") pod \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") "
Oct 30 03:14:55.041681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.041067    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-home\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-home\") pod \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") "
Oct 30 03:14:55.041681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.041103    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-dshm\") pod \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") "
Oct 30 03:14:55.041681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.041189    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tmp-dir\") pod \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") "
Oct 30 03:14:55.041681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.041288    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-model-cache\") pod \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\" (UID: \"80e943c9-b7ad-4d36-866d-ae0fe597fca1\") "
Oct 30 03:14:55.042558 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.042527    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-model-cache" pod "80e943c9-b7ad-4d36-866d-ae0fe597fca1" (UID: "80e943c9-b7ad-4d36-866d-ae0fe597fca1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:55.042988 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.042963    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-home" pod "80e943c9-b7ad-4d36-866d-ae0fe597fca1" (UID: "80e943c9-b7ad-4d36-866d-ae0fe597fca1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:55.046023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.045753    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tls-certs" pod "80e943c9-b7ad-4d36-866d-ae0fe597fca1" (UID: "80e943c9-b7ad-4d36-866d-ae0fe597fca1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:14:55.046399 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.046366    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e943c9-b7ad-4d36-866d-ae0fe597fca1-kube-api-access-7dwtf" pod "80e943c9-b7ad-4d36-866d-ae0fe597fca1" (UID: "80e943c9-b7ad-4d36-866d-ae0fe597fca1"). InnerVolumeSpecName "kube-api-access-7dwtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:55.046857 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.046832    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tmp-dir" pod "80e943c9-b7ad-4d36-866d-ae0fe597fca1" (UID: "80e943c9-b7ad-4d36-866d-ae0fe597fca1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:55.053529 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.051560    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-dshm" pod "80e943c9-b7ad-4d36-866d-ae0fe597fca1" (UID: "80e943c9-b7ad-4d36-866d-ae0fe597fca1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:55.144818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.144779    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:55.144818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.144813    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dwtf\" (UniqueName: \"kubernetes.io/projected/80e943c9-b7ad-4d36-866d-ae0fe597fca1-kube-api-access-7dwtf\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:55.144818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.144823    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:55.145070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.144831    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:55.145070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.144839    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:55.145070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.144849    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e943c9-b7ad-4d36-866d-ae0fe597fca1-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:14:55.843353 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.843318    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"
Oct 30 03:14:55.843596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.843375    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt" event={"ID":"80e943c9-b7ad-4d36-866d-ae0fe597fca1","Type":"ContainerDied","Data":"1240733cb6f97603aa658ede1b3622bc6ed3f152b03ca3b1b3b155eda9e7d7d5"}
Oct 30 03:14:55.843596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.843443    4278 scope.go:122] "RemoveContainer" containerID="53c6e3527fff38108492f9004b87e4d12081da31ff5211d02d3e1fac390e8829"
Oct 30 03:14:55.909956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.909911    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"]
Oct 30 03:14:55.923346 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:55.923307    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/auth-test-model-360818-kserve-6ffc4748f-wbpnt"]
Oct 30 03:14:56.201361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:14:56.201268    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e943c9-b7ad-4d36-866d-ae0fe597fca1" path="/var/lib/kubelet/pods/80e943c9-b7ad-4d36-866d-ae0fe597fca1/volumes"
Oct 30 03:15:00.054816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.054769    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"]
Oct 30 03:15:00.055637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.055607    4278 cpu_manager.go:414] "containerMap: removing container" podUID="80e943c9-b7ad-4d36-866d-ae0fe597fca1" containerName="main"
Oct 30 03:15:00.055819 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.055795    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e943c9-b7ad-4d36-866d-ae0fe597fca1" containerName="main"
Oct 30 03:15:00.055944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.055861    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f66096de-fc52-463a-a740-09bc1a1692d0" containerName="main"
Oct 30 03:15:00.055944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.055868    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66096de-fc52-463a-a740-09bc1a1692d0" containerName="main"
Oct 30 03:15:00.056073 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.056059    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e943c9-b7ad-4d36-866d-ae0fe597fca1" containerName="main"
Oct 30 03:15:00.056133 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.056081    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66096de-fc52-463a-a740-09bc1a1692d0" containerName="main"
Oct 30 03:15:00.060132 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.060103    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.080415 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.080378    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"]
Oct 30 03:15:00.085996 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.085964    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"openshift-service-ca.crt\""
Oct 30 03:15:00.086284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.086128    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"kube-root-ca.crt\""
Oct 30 03:15:00.086393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.086314    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"default-dockercfg-ml4hg\""
Oct 30 03:15:00.089002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.088967    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"sub-test-model-9245d1-kserve-self-signed-certs\""
Oct 30 03:15:00.130717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.130674    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpsls\" (UniqueName: \"kubernetes.io/projected/4e362931-63ca-4a40-8c39-2725e38cdc3d-kube-api-access-tpsls\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.130717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.130730    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-home\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.131011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.130930    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e362931-63ca-4a40-8c39-2725e38cdc3d-tls-certs\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.131131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.131113    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-model-cache\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.131232 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.131217    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-dshm\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.131298 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.131287    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-tmp-dir\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.148485 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.148412    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"]
Oct 30 03:15:00.167648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.167606    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"
Oct 30 03:15:00.179453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.175628    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"]
Oct 30 03:15:00.198781 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.198746    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:15:00.199939 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.199907    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.268087    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpsls\" (UniqueName: \"kubernetes.io/projected/4e362931-63ca-4a40-8c39-2725e38cdc3d-kube-api-access-tpsls\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.268151    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-home\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.269535    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e362931-63ca-4a40-8c39-2725e38cdc3d-tls-certs\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.269663    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6\" (UID: \"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.269730    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjpw\" (UniqueName: \"kubernetes.io/projected/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-kube-api-access-xgjpw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6\" (UID: \"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.269783    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5968a2a-86ba-4717-8f0f-df111b79f123-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf\" (UID: \"c5968a2a-86ba-4717-8f0f-df111b79f123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.269814    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-model-cache\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.269844    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnsbw\" (UniqueName: \"kubernetes.io/projected/c5968a2a-86ba-4717-8f0f-df111b79f123-kube-api-access-dnsbw\") pod \"maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf\" (UID: \"c5968a2a-86ba-4717-8f0f-df111b79f123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.269988    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-dshm\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.270114    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-tmp-dir\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.272704    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-tmp-dir\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.273963    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"]
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.274044    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-home\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.284449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.274354    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-model-cache\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.311491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.310883    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"]
Oct 30 03:15:00.311491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.310914    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"
Oct 30 03:15:00.319365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.317608    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"]
Oct 30 03:15:00.320334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.319972    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"
Oct 30 03:15:00.321576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.321484    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e362931-63ca-4a40-8c39-2725e38cdc3d-tls-certs\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.330074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.322977    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"
Oct 30 03:15:00.330074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.329016    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-dshm\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.350464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.333463    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpsls\" (UniqueName: \"kubernetes.io/projected/4e362931-63ca-4a40-8c39-2725e38cdc3d-kube-api-access-tpsls\") pod \"sub-test-model-9245d1-kserve-655d677855-9kwmk\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.385474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.380642    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6\" (UID: \"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"
Oct 30 03:15:00.385474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.380747    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjpw\" (UniqueName: \"kubernetes.io/projected/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-kube-api-access-xgjpw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6\" (UID: \"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"
Oct 30 03:15:00.385474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.380801    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5968a2a-86ba-4717-8f0f-df111b79f123-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf\" (UID: \"c5968a2a-86ba-4717-8f0f-df111b79f123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"
Oct 30 03:15:00.385474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.380837    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnsbw\" (UniqueName: \"kubernetes.io/projected/c5968a2a-86ba-4717-8f0f-df111b79f123-kube-api-access-dnsbw\") pod \"maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf\" (UID: \"c5968a2a-86ba-4717-8f0f-df111b79f123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"
Oct 30 03:15:00.385474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.381275    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:00.385474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.382388    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5968a2a-86ba-4717-8f0f-df111b79f123-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf\" (UID: \"c5968a2a-86ba-4717-8f0f-df111b79f123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"
Oct 30 03:15:00.385474 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.383077    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6\" (UID: \"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"
Oct 30 03:15:00.410074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.396916    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"]
Oct 30 03:15:00.423932 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.415953    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"]
Oct 30 03:15:00.462379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.452718    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnsbw\" (UniqueName: \"kubernetes.io/projected/c5968a2a-86ba-4717-8f0f-df111b79f123-kube-api-access-dnsbw\") pod \"maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf\" (UID: \"c5968a2a-86ba-4717-8f0f-df111b79f123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"
Oct 30 03:15:00.462379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.461337    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjpw\" (UniqueName: \"kubernetes.io/projected/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-kube-api-access-xgjpw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6\" (UID: \"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"
Oct 30 03:15:00.567523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.524506    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"]
Oct 30 03:15:00.599703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.595584    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmb6\" (UniqueName: \"kubernetes.io/projected/ea6c297a-e973-4e59-859f-5c4862900275-kube-api-access-6tmb6\") pod \"maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x\" (UID: \"ea6c297a-e973-4e59-859f-5c4862900275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"
Oct 30 03:15:00.599703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.595659    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e75f3b5-4d25-4fed-91f1-99a41800f320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws\" (UID: \"9e75f3b5-4d25-4fed-91f1-99a41800f320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"
Oct 30 03:15:00.599703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.595940    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvth\" (UniqueName: \"kubernetes.io/projected/bfefef3e-400b-48d3-b4f8-bf432ee80c29-kube-api-access-ffvth\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx\" (UID: \"bfefef3e-400b-48d3-b4f8-bf432ee80c29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"
Oct 30 03:15:00.599703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.596051    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfefef3e-400b-48d3-b4f8-bf432ee80c29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx\" (UID: \"bfefef3e-400b-48d3-b4f8-bf432ee80c29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"
Oct 30 03:15:00.599703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.596121    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6c297a-e973-4e59-859f-5c4862900275-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x\" (UID: \"ea6c297a-e973-4e59-859f-5c4862900275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"
Oct 30 03:15:00.599703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.596495    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7lf\" (UniqueName: \"kubernetes.io/projected/9e75f3b5-4d25-4fed-91f1-99a41800f320-kube-api-access-4p7lf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws\" (UID: \"9e75f3b5-4d25-4fed-91f1-99a41800f320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"
Oct 30 03:15:00.599703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.597002    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"
Oct 30 03:15:00.656447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.614663    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"
Oct 30 03:15:00.656447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.652401    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"]
Oct 30 03:15:00.688953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.684724    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"]
Oct 30 03:15:00.688953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.687946    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"
Oct 30 03:15:00.698556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.698489    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvth\" (UniqueName: \"kubernetes.io/projected/bfefef3e-400b-48d3-b4f8-bf432ee80c29-kube-api-access-ffvth\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx\" (UID: \"bfefef3e-400b-48d3-b4f8-bf432ee80c29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"
Oct 30 03:15:00.698711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.698561    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfefef3e-400b-48d3-b4f8-bf432ee80c29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx\" (UID: \"bfefef3e-400b-48d3-b4f8-bf432ee80c29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"
Oct 30 03:15:00.698711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.698618    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6c297a-e973-4e59-859f-5c4862900275-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x\" (UID: \"ea6c297a-e973-4e59-859f-5c4862900275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"
Oct 30 03:15:00.698854 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.698807    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7lf\" (UniqueName: \"kubernetes.io/projected/9e75f3b5-4d25-4fed-91f1-99a41800f320-kube-api-access-4p7lf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws\" (UID: \"9e75f3b5-4d25-4fed-91f1-99a41800f320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"
Oct 30 03:15:00.699054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.698945    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmb6\" (UniqueName: \"kubernetes.io/projected/ea6c297a-e973-4e59-859f-5c4862900275-kube-api-access-6tmb6\") pod \"maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x\" (UID: \"ea6c297a-e973-4e59-859f-5c4862900275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"
Oct 30 03:15:00.699054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.698996    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e75f3b5-4d25-4fed-91f1-99a41800f320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws\" (UID: \"9e75f3b5-4d25-4fed-91f1-99a41800f320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"
Oct 30 03:15:00.700026 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.699939    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e75f3b5-4d25-4fed-91f1-99a41800f320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws\" (UID: \"9e75f3b5-4d25-4fed-91f1-99a41800f320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"
Oct 30 03:15:00.700820 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.700788    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"]
Oct 30 03:15:00.701389 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.701362    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6c297a-e973-4e59-859f-5c4862900275-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x\" (UID: \"ea6c297a-e973-4e59-859f-5c4862900275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"
Oct 30 03:15:00.702003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.701947    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"
Oct 30 03:15:00.719839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.718129    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfefef3e-400b-48d3-b4f8-bf432ee80c29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx\" (UID: \"bfefef3e-400b-48d3-b4f8-bf432ee80c29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"
Oct 30 03:15:00.719839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.718495    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"]
Oct 30 03:15:00.719839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.718536    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"]
Oct 30 03:15:00.719839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.718565    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"]
Oct 30 03:15:00.719839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.718773    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"
Oct 30 03:15:00.724319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.724292    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"]
Oct 30 03:15:00.725815 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.725640    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"
Oct 30 03:15:00.731068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.731040    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7lf\" (UniqueName: \"kubernetes.io/projected/9e75f3b5-4d25-4fed-91f1-99a41800f320-kube-api-access-4p7lf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws\" (UID: \"9e75f3b5-4d25-4fed-91f1-99a41800f320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"
Oct 30 03:15:00.737489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.736596    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"]
Oct 30 03:15:00.737489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.736636    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"]
Oct 30 03:15:00.737489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.736663    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"]
Oct 30 03:15:00.737489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.736763    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"
Oct 30 03:15:00.740101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.740077    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmb6\" (UniqueName: \"kubernetes.io/projected/ea6c297a-e973-4e59-859f-5c4862900275-kube-api-access-6tmb6\") pod \"maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x\" (UID: \"ea6c297a-e973-4e59-859f-5c4862900275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"
Oct 30 03:15:00.743719 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.743696    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"]
Oct 30 03:15:00.763802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.762813    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"]
Oct 30 03:15:00.774941 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.774909    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"]
Oct 30 03:15:00.775216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.775190    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"
Oct 30 03:15:00.793546 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.793510    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"]
Oct 30 03:15:00.799760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.799724    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"]
Oct 30 03:15:00.800082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.800066    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"
Oct 30 03:15:00.800359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.800344    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"
Oct 30 03:15:00.801992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.781464    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvth\" (UniqueName: \"kubernetes.io/projected/bfefef3e-400b-48d3-b4f8-bf432ee80c29-kube-api-access-ffvth\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx\" (UID: \"bfefef3e-400b-48d3-b4f8-bf432ee80c29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"
Oct 30 03:15:00.802843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.802816    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c6501-88c9-4285-9b60-2d38d788bbd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4\" (UID: \"969c6501-88c9-4285-9b60-2d38d788bbd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"
Oct 30 03:15:00.803068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.803047    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xbn\" (UniqueName: \"kubernetes.io/projected/969c6501-88c9-4285-9b60-2d38d788bbd6-kube-api-access-f6xbn\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4\" (UID: \"969c6501-88c9-4285-9b60-2d38d788bbd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"
Oct 30 03:15:00.803332 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.803313    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kq4z\" (UniqueName: \"kubernetes.io/projected/fed730a6-41b7-4cfc-af55-ae03dbc48d38-kube-api-access-5kq4z\") pod \"maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws\" (UID: \"fed730a6-41b7-4cfc-af55-ae03dbc48d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"
Oct 30 03:15:00.803668 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.803542    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6f2b85-9672-4fc7-8a8a-0393e7886187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb\" (UID: \"8e6f2b85-9672-4fc7-8a8a-0393e7886187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"
Oct 30 03:15:00.803852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.803831    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xznh\" (UniqueName: \"kubernetes.io/projected/6c90ec76-2fe7-40e2-b88b-8b766b28e158-kube-api-access-8xznh\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc\" (UID: \"6c90ec76-2fe7-40e2-b88b-8b766b28e158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"
Oct 30 03:15:00.805163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.805142    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ftxp\" (UniqueName: \"kubernetes.io/projected/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-kube-api-access-2ftxp\") pod \"maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j\" (UID: \"5103b31a-7eaf-4364-9af6-1b7eb34e1d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"
Oct 30 03:15:00.805488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.805469    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzjx\" (UniqueName: \"kubernetes.io/projected/8e6f2b85-9672-4fc7-8a8a-0393e7886187-kube-api-access-7gzjx\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb\" (UID: \"8e6f2b85-9672-4fc7-8a8a-0393e7886187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"
Oct 30 03:15:00.805791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.805773    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j\" (UID: \"5103b31a-7eaf-4364-9af6-1b7eb34e1d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"
Oct 30 03:15:00.806007 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.805988    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c90ec76-2fe7-40e2-b88b-8b766b28e158-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc\" (UID: \"6c90ec76-2fe7-40e2-b88b-8b766b28e158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"
Oct 30 03:15:00.806176 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.806160    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed730a6-41b7-4cfc-af55-ae03dbc48d38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws\" (UID: \"fed730a6-41b7-4cfc-af55-ae03dbc48d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"
Oct 30 03:15:00.814333 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.814308    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"]
Oct 30 03:15:00.872045 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.871955    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"]
Oct 30 03:15:00.886871 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.886771    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"]
Oct 30 03:15:00.903216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.903010    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"
Oct 30 03:15:00.921392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.921357    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xznh\" (UniqueName: \"kubernetes.io/projected/6c90ec76-2fe7-40e2-b88b-8b766b28e158-kube-api-access-8xznh\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc\" (UID: \"6c90ec76-2fe7-40e2-b88b-8b766b28e158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"
Oct 30 03:15:00.921891 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.921871    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ftxp\" (UniqueName: \"kubernetes.io/projected/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-kube-api-access-2ftxp\") pod \"maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j\" (UID: \"5103b31a-7eaf-4364-9af6-1b7eb34e1d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"
Oct 30 03:15:00.922030 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.922019    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzjx\" (UniqueName: \"kubernetes.io/projected/8e6f2b85-9672-4fc7-8a8a-0393e7886187-kube-api-access-7gzjx\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb\" (UID: \"8e6f2b85-9672-4fc7-8a8a-0393e7886187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"
Oct 30 03:15:00.922113 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.922101    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg58m\" (UniqueName: \"kubernetes.io/projected/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-kube-api-access-vg58m\") pod \"maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj\" (UID: \"8223ce14-326f-40bd-bb96-dc1b24bb4fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"
Oct 30 03:15:00.922211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.922201    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02df413-cc1b-4d7e-8209-16022147c7a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7\" (UID: \"a02df413-cc1b-4d7e-8209-16022147c7a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"
Oct 30 03:15:00.922287 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.922278    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hc7s\" (UniqueName: \"kubernetes.io/projected/0d2f4868-04fa-430b-93bc-9644c7a526cd-kube-api-access-9hc7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb\" (UID: \"0d2f4868-04fa-430b-93bc-9644c7a526cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"
Oct 30 03:15:00.922366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.922358    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j\" (UID: \"5103b31a-7eaf-4364-9af6-1b7eb34e1d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"
Oct 30 03:15:00.922467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.922457    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gzz\" (UniqueName: \"kubernetes.io/projected/a02df413-cc1b-4d7e-8209-16022147c7a1-kube-api-access-j8gzz\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7\" (UID: \"a02df413-cc1b-4d7e-8209-16022147c7a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"
Oct 30 03:15:00.922644 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.922630    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c90ec76-2fe7-40e2-b88b-8b766b28e158-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc\" (UID: \"6c90ec76-2fe7-40e2-b88b-8b766b28e158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"
Oct 30 03:15:00.922718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.922710    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed730a6-41b7-4cfc-af55-ae03dbc48d38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws\" (UID: \"fed730a6-41b7-4cfc-af55-ae03dbc48d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"
Oct 30 03:15:00.922863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.922854    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2f4868-04fa-430b-93bc-9644c7a526cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb\" (UID: \"0d2f4868-04fa-430b-93bc-9644c7a526cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"
Oct 30 03:15:00.922946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.922937    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c6501-88c9-4285-9b60-2d38d788bbd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4\" (UID: \"969c6501-88c9-4285-9b60-2d38d788bbd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"
Oct 30 03:15:00.923033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.923025    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xbn\" (UniqueName: \"kubernetes.io/projected/969c6501-88c9-4285-9b60-2d38d788bbd6-kube-api-access-f6xbn\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4\" (UID: \"969c6501-88c9-4285-9b60-2d38d788bbd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"
Oct 30 03:15:00.923108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.923100    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj\" (UID: \"8223ce14-326f-40bd-bb96-dc1b24bb4fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"
Oct 30 03:15:00.923219 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.923207    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kq4z\" (UniqueName: \"kubernetes.io/projected/fed730a6-41b7-4cfc-af55-ae03dbc48d38-kube-api-access-5kq4z\") pod \"maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws\" (UID: \"fed730a6-41b7-4cfc-af55-ae03dbc48d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"
Oct 30 03:15:00.923292 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.923283    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6f2b85-9672-4fc7-8a8a-0393e7886187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb\" (UID: \"8e6f2b85-9672-4fc7-8a8a-0393e7886187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"
Oct 30 03:15:00.927500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.924880    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6f2b85-9672-4fc7-8a8a-0393e7886187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb\" (UID: \"8e6f2b85-9672-4fc7-8a8a-0393e7886187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"
Oct 30 03:15:00.927655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.926821    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c6501-88c9-4285-9b60-2d38d788bbd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4\" (UID: \"969c6501-88c9-4285-9b60-2d38d788bbd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"
Oct 30 03:15:00.931288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.931264    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c90ec76-2fe7-40e2-b88b-8b766b28e158-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc\" (UID: \"6c90ec76-2fe7-40e2-b88b-8b766b28e158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"
Oct 30 03:15:00.933823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.933501    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j\" (UID: \"5103b31a-7eaf-4364-9af6-1b7eb34e1d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"
Oct 30 03:15:00.938451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.938080    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed730a6-41b7-4cfc-af55-ae03dbc48d38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws\" (UID: \"fed730a6-41b7-4cfc-af55-ae03dbc48d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"
Oct 30 03:15:00.946833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.946797    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"
Oct 30 03:15:00.974320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.973539    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xznh\" (UniqueName: \"kubernetes.io/projected/6c90ec76-2fe7-40e2-b88b-8b766b28e158-kube-api-access-8xznh\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc\" (UID: \"6c90ec76-2fe7-40e2-b88b-8b766b28e158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"
Oct 30 03:15:00.974320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.973566    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xbn\" (UniqueName: \"kubernetes.io/projected/969c6501-88c9-4285-9b60-2d38d788bbd6-kube-api-access-f6xbn\") pod \"maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4\" (UID: \"969c6501-88c9-4285-9b60-2d38d788bbd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"
Oct 30 03:15:00.988566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.979292    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzjx\" (UniqueName: \"kubernetes.io/projected/8e6f2b85-9672-4fc7-8a8a-0393e7886187-kube-api-access-7gzjx\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb\" (UID: \"8e6f2b85-9672-4fc7-8a8a-0393e7886187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"
Oct 30 03:15:00.988566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:00.983917    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ftxp\" (UniqueName: \"kubernetes.io/projected/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-kube-api-access-2ftxp\") pod \"maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j\" (UID: \"5103b31a-7eaf-4364-9af6-1b7eb34e1d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"
Oct 30 03:15:01.006147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.006039    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kq4z\" (UniqueName: \"kubernetes.io/projected/fed730a6-41b7-4cfc-af55-ae03dbc48d38-kube-api-access-5kq4z\") pod \"maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws\" (UID: \"fed730a6-41b7-4cfc-af55-ae03dbc48d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"
Oct 30 03:15:01.006484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.006410    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"
Oct 30 03:15:01.024165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.024137    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"
Oct 30 03:15:01.025690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.024855    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02df413-cc1b-4d7e-8209-16022147c7a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7\" (UID: \"a02df413-cc1b-4d7e-8209-16022147c7a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"
Oct 30 03:15:01.025690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.024912    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hc7s\" (UniqueName: \"kubernetes.io/projected/0d2f4868-04fa-430b-93bc-9644c7a526cd-kube-api-access-9hc7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb\" (UID: \"0d2f4868-04fa-430b-93bc-9644c7a526cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"
Oct 30 03:15:01.025690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.025003    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gzz\" (UniqueName: \"kubernetes.io/projected/a02df413-cc1b-4d7e-8209-16022147c7a1-kube-api-access-j8gzz\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7\" (UID: \"a02df413-cc1b-4d7e-8209-16022147c7a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"
Oct 30 03:15:01.025690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.025123    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2f4868-04fa-430b-93bc-9644c7a526cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb\" (UID: \"0d2f4868-04fa-430b-93bc-9644c7a526cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"
Oct 30 03:15:01.025690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.025216    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj\" (UID: \"8223ce14-326f-40bd-bb96-dc1b24bb4fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"
Oct 30 03:15:01.027206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.025990    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg58m\" (UniqueName: \"kubernetes.io/projected/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-kube-api-access-vg58m\") pod \"maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj\" (UID: \"8223ce14-326f-40bd-bb96-dc1b24bb4fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"
Oct 30 03:15:01.027206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.026261    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2f4868-04fa-430b-93bc-9644c7a526cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb\" (UID: \"0d2f4868-04fa-430b-93bc-9644c7a526cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"
Oct 30 03:15:01.027206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.026981    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj\" (UID: \"8223ce14-326f-40bd-bb96-dc1b24bb4fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"
Oct 30 03:15:01.027206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.027165    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02df413-cc1b-4d7e-8209-16022147c7a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7\" (UID: \"a02df413-cc1b-4d7e-8209-16022147c7a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"
Oct 30 03:15:01.066715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.064932    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gzz\" (UniqueName: \"kubernetes.io/projected/a02df413-cc1b-4d7e-8209-16022147c7a1-kube-api-access-j8gzz\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7\" (UID: \"a02df413-cc1b-4d7e-8209-16022147c7a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"
Oct 30 03:15:01.066715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.065620    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg58m\" (UniqueName: \"kubernetes.io/projected/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-kube-api-access-vg58m\") pod \"maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj\" (UID: \"8223ce14-326f-40bd-bb96-dc1b24bb4fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"
Oct 30 03:15:01.069517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.068100    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hc7s\" (UniqueName: \"kubernetes.io/projected/0d2f4868-04fa-430b-93bc-9644c7a526cd-kube-api-access-9hc7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb\" (UID: \"0d2f4868-04fa-430b-93bc-9644c7a526cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"
Oct 30 03:15:01.123009 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.108149    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"]
Oct 30 03:15:01.143255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.141708    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"
Oct 30 03:15:01.167849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.166687    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"
Oct 30 03:15:01.172064 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.172035    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"
Oct 30 03:15:01.177002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.176798    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"
Oct 30 03:15:01.207649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.207615    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"
Oct 30 03:15:01.277068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.272343    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"
Oct 30 03:15:01.302755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.302645    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"
Oct 30 03:15:01.480667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.480633    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"]
Oct 30 03:15:01.498161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.498135    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"]
Oct 30 03:15:01.498665 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:15:01.498645    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1d34d8_4d62_4a52_bb97_ae9ea24eac00.slice/crio-9903b4689f2e50514393d00c46c775069e384f58976b00fc7e149b117e5c9511 WatchSource:0}: Error finding container 9903b4689f2e50514393d00c46c775069e384f58976b00fc7e149b117e5c9511: Status 404 returned error can't find the container with id 9903b4689f2e50514393d00c46c775069e384f58976b00fc7e149b117e5c9511
Oct 30 03:15:01.572767 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:15:01.572722    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e75f3b5_4d25_4fed_91f1_99a41800f320.slice/crio-97791a83b325a8da09ec335c1f86272ab9ae2b30adf8295bc4575915baf1a2ca WatchSource:0}: Error finding container 97791a83b325a8da09ec335c1f86272ab9ae2b30adf8295bc4575915baf1a2ca: Status 404 returned error can't find the container with id 97791a83b325a8da09ec335c1f86272ab9ae2b30adf8295bc4575915baf1a2ca
Oct 30 03:15:01.573399 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.573366    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"]
Oct 30 03:15:01.597664 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:15:01.597625    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfefef3e_400b_48d3_b4f8_bf432ee80c29.slice/crio-16d7613c04dfcce476aea18f17a44ba11e711e884c3ef6d776bcaf5d3b821ba7 WatchSource:0}: Error finding container 16d7613c04dfcce476aea18f17a44ba11e711e884c3ef6d776bcaf5d3b821ba7: Status 404 returned error can't find the container with id 16d7613c04dfcce476aea18f17a44ba11e711e884c3ef6d776bcaf5d3b821ba7
Oct 30 03:15:01.603481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.603448    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"]
Oct 30 03:15:01.635011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.633704    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"]
Oct 30 03:15:01.653738 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:15:01.653702    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6f2b85_9672_4fc7_8a8a_0393e7886187.slice/crio-f00d0c0ec3aa9b04e4f322aeb61b062e5a2406ce23c68a3186ee2d3236063e7e WatchSource:0}: Error finding container f00d0c0ec3aa9b04e4f322aeb61b062e5a2406ce23c68a3186ee2d3236063e7e: Status 404 returned error can't find the container with id f00d0c0ec3aa9b04e4f322aeb61b062e5a2406ce23c68a3186ee2d3236063e7e
Oct 30 03:15:01.729282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.729257    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"]
Oct 30 03:15:01.737589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.737001    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"]
Oct 30 03:15:01.767616 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:15:01.767576    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed730a6_41b7_4cfc_af55_ae03dbc48d38.slice/crio-f8eff2a05b5745a865ba552e82f2bdb86cbcbdd91ea40d7c657d0f438ea08e3d WatchSource:0}: Error finding container f8eff2a05b5745a865ba552e82f2bdb86cbcbdd91ea40d7c657d0f438ea08e3d: Status 404 returned error can't find the container with id f8eff2a05b5745a865ba552e82f2bdb86cbcbdd91ea40d7c657d0f438ea08e3d
Oct 30 03:15:01.783547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.783520    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"]
Oct 30 03:15:01.788383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.788356    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"]
Oct 30 03:15:01.803112 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:15:01.803070    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5103b31a_7eaf_4364_9af6_1b7eb34e1d40.slice/crio-b01dd662818b22078535dd5d152dbf11b722d310c76768b428133b733ecd036f WatchSource:0}: Error finding container b01dd662818b22078535dd5d152dbf11b722d310c76768b428133b733ecd036f: Status 404 returned error can't find the container with id b01dd662818b22078535dd5d152dbf11b722d310c76768b428133b733ecd036f
Oct 30 03:15:01.803994 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:15:01.803960    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969c6501_88c9_4285_9b60_2d38d788bbd6.slice/crio-f6a91ce9fabbb54384984de3e9380b34576bb28567736ccf5194ae0569edf742 WatchSource:0}: Error finding container f6a91ce9fabbb54384984de3e9380b34576bb28567736ccf5194ae0569edf742: Status 404 returned error can't find the container with id f6a91ce9fabbb54384984de3e9380b34576bb28567736ccf5194ae0569edf742
Oct 30 03:15:01.873020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.871526    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"]
Oct 30 03:15:01.877655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.877616    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"]
Oct 30 03:15:01.904445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.904401    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"]
Oct 30 03:15:01.908187 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.908155    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" event={"ID":"6c90ec76-2fe7-40e2-b88b-8b766b28e158","Type":"ContainerStarted","Data":"f1d1f2fad2b3dd124185a27ee36ce57d7cfb4e2de99e1f18c31da475817a2d00"}
Oct 30 03:15:01.909772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.909742    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws" event={"ID":"9e75f3b5-4d25-4fed-91f1-99a41800f320","Type":"ContainerStarted","Data":"97791a83b325a8da09ec335c1f86272ab9ae2b30adf8295bc4575915baf1a2ca"}
Oct 30 03:15:01.911136 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.911113    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" event={"ID":"8e6f2b85-9672-4fc7-8a8a-0393e7886187","Type":"ContainerStarted","Data":"f00d0c0ec3aa9b04e4f322aeb61b062e5a2406ce23c68a3186ee2d3236063e7e"}
Oct 30 03:15:01.912910 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.912878    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk" event={"ID":"4e362931-63ca-4a40-8c39-2725e38cdc3d","Type":"ContainerStarted","Data":"25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135"}
Oct 30 03:15:01.912991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.912921    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk" event={"ID":"4e362931-63ca-4a40-8c39-2725e38cdc3d","Type":"ContainerStarted","Data":"ce8f1a650aa7bb70aefb243d084f492406cb21661134a014a1c6ca2cd395a795"}
Oct 30 03:15:01.914268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.914245    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" event={"ID":"fed730a6-41b7-4cfc-af55-ae03dbc48d38","Type":"ContainerStarted","Data":"f8eff2a05b5745a865ba552e82f2bdb86cbcbdd91ea40d7c657d0f438ea08e3d"}
Oct 30 03:15:01.915722 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.915690    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6" event={"ID":"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00","Type":"ContainerStarted","Data":"9903b4689f2e50514393d00c46c775069e384f58976b00fc7e149b117e5c9511"}
Oct 30 03:15:01.916976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.916957    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" event={"ID":"969c6501-88c9-4285-9b60-2d38d788bbd6","Type":"ContainerStarted","Data":"f6a91ce9fabbb54384984de3e9380b34576bb28567736ccf5194ae0569edf742"}
Oct 30 03:15:01.918297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.918277    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx" event={"ID":"bfefef3e-400b-48d3-b4f8-bf432ee80c29","Type":"ContainerStarted","Data":"16d7613c04dfcce476aea18f17a44ba11e711e884c3ef6d776bcaf5d3b821ba7"}
Oct 30 03:15:01.919755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.919728    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" event={"ID":"ea6c297a-e973-4e59-859f-5c4862900275","Type":"ContainerStarted","Data":"f5723e36a9c24219e59280c4ba759a6653350df170e0c0b0a1e1b293227f06cb"}
Oct 30 03:15:01.921724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.921700    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" event={"ID":"c5968a2a-86ba-4717-8f0f-df111b79f123","Type":"ContainerStarted","Data":"4be2cdceafac7b6d86ad7f3ef738d9f27d2504d18b397701ae2050ef299f2746"}
Oct 30 03:15:01.923260 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.923228    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" event={"ID":"5103b31a-7eaf-4364-9af6-1b7eb34e1d40","Type":"ContainerStarted","Data":"b01dd662818b22078535dd5d152dbf11b722d310c76768b428133b733ecd036f"}
Oct 30 03:15:01.924701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.924678    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7" event={"ID":"a02df413-cc1b-4d7e-8209-16022147c7a1","Type":"ContainerStarted","Data":"18746edfbb9efc4aea4be6a80ef9285186d1711abebb58c60c902c461e0e042f"}
Oct 30 03:15:01.926119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.926098    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"]
Oct 30 03:15:01.954783 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:01.954712    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk" podStartSLOduration=1.9546953710000001 podStartE2EDuration="1.954695371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:01.95415749 +0000 UTC m=+2039.878365677" watchObservedRunningTime="2026-10-30 03:15:01.954695371 +0000 UTC m=+2039.878903563"
Oct 30 03:15:01.976241 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:15:01.976209    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8223ce14_326f_40bd_bb96_dc1b24bb4fd0.slice/crio-48cff14b32d51b0c171802dfa504a916d402f40363105b4e79916c6a94548885 WatchSource:0}: Error finding container 48cff14b32d51b0c171802dfa504a916d402f40363105b4e79916c6a94548885: Status 404 returned error can't find the container with id 48cff14b32d51b0c171802dfa504a916d402f40363105b4e79916c6a94548885
Oct 30 03:15:02.062656 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:15:02.062616    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2f4868_04fa_430b_93bc_9644c7a526cd.slice/crio-4eab06c511e03b8608acf1fbe7db2f5e881a2d0c98e8ed8d6403bf11d0051ed6 WatchSource:0}: Error finding container 4eab06c511e03b8608acf1fbe7db2f5e881a2d0c98e8ed8d6403bf11d0051ed6: Status 404 returned error can't find the container with id 4eab06c511e03b8608acf1fbe7db2f5e881a2d0c98e8ed8d6403bf11d0051ed6
Oct 30 03:15:02.384649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.384609    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:02.935390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.935298    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" event={"ID":"fed730a6-41b7-4cfc-af55-ae03dbc48d38","Type":"ContainerStarted","Data":"b5c315e80fe884ef34046479878b5b99ab00aaf94028e2068afcffde4ac96e46"}
Oct 30 03:15:02.937197 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.937166    4278 generic.go:356] "Generic (PLEG): container finished" podID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerID="cfffb6ecd772ef1143711ddc1645ba0c7b005bb6f0dc05116e6ce56f151b73a8" exitCode=60
Oct 30 03:15:02.937333 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.937238    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6" event={"ID":"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00","Type":"ContainerDied","Data":"cfffb6ecd772ef1143711ddc1645ba0c7b005bb6f0dc05116e6ce56f151b73a8"}
Oct 30 03:15:02.937598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.937574    4278 scope.go:122] "RemoveContainer" containerID="cfffb6ecd772ef1143711ddc1645ba0c7b005bb6f0dc05116e6ce56f151b73a8"
Oct 30 03:15:02.938899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.938869    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" event={"ID":"969c6501-88c9-4285-9b60-2d38d788bbd6","Type":"ContainerStarted","Data":"bd547fbae002c94c379ce4443b8fd1568ac512afe2faf0645c6fbb6eeb0d4e4c"}
Oct 30 03:15:02.942312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.941440    4278 generic.go:356] "Generic (PLEG): container finished" podID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerID="244e5c0b29d8e4b772a43c10b6a4599e5a4f7fa5f9aa4b224a9d6efa182d1496" exitCode=60
Oct 30 03:15:02.942312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.941445    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx" event={"ID":"bfefef3e-400b-48d3-b4f8-bf432ee80c29","Type":"ContainerDied","Data":"244e5c0b29d8e4b772a43c10b6a4599e5a4f7fa5f9aa4b224a9d6efa182d1496"}
Oct 30 03:15:02.942312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.941801    4278 scope.go:122] "RemoveContainer" containerID="244e5c0b29d8e4b772a43c10b6a4599e5a4f7fa5f9aa4b224a9d6efa182d1496"
Oct 30 03:15:02.943533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.943509    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" event={"ID":"ea6c297a-e973-4e59-859f-5c4862900275","Type":"ContainerStarted","Data":"e2ab0fd0b596fe7f494693f16bea01a11ab65789fb585b0079414f00084e6de2"}
Oct 30 03:15:02.945284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.945256    4278 generic.go:356] "Generic (PLEG): container finished" podID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerID="07ab3ff6de4e6d77a3b2f561aed388cf2901f563d5b414437de891085b6830e0" exitCode=60
Oct 30 03:15:02.945408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.945382    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" event={"ID":"c5968a2a-86ba-4717-8f0f-df111b79f123","Type":"ContainerDied","Data":"07ab3ff6de4e6d77a3b2f561aed388cf2901f563d5b414437de891085b6830e0"}
Oct 30 03:15:02.945676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.945659    4278 scope.go:122] "RemoveContainer" containerID="07ab3ff6de4e6d77a3b2f561aed388cf2901f563d5b414437de891085b6830e0"
Oct 30 03:15:02.947192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.947165    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" event={"ID":"8223ce14-326f-40bd-bb96-dc1b24bb4fd0","Type":"ContainerStarted","Data":"def5c559f06cef1b961e32ebd390d00300599a548928bb02fecfbfd24e06e299"}
Oct 30 03:15:02.947302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.947220    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" event={"ID":"8223ce14-326f-40bd-bb96-dc1b24bb4fd0","Type":"ContainerStarted","Data":"48cff14b32d51b0c171802dfa504a916d402f40363105b4e79916c6a94548885"}
Oct 30 03:15:02.948879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.948853    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" event={"ID":"5103b31a-7eaf-4364-9af6-1b7eb34e1d40","Type":"ContainerStarted","Data":"c66da6417ba6604f6b3b1b5a089d4a32510786bb5a56c04849889b553eb51d31"}
Oct 30 03:15:02.950338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.950270    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" event={"ID":"0d2f4868-04fa-430b-93bc-9644c7a526cd","Type":"ContainerStarted","Data":"4eab06c511e03b8608acf1fbe7db2f5e881a2d0c98e8ed8d6403bf11d0051ed6"}
Oct 30 03:15:02.953620 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.953592    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" event={"ID":"6c90ec76-2fe7-40e2-b88b-8b766b28e158","Type":"ContainerStarted","Data":"818eb5259fed8daa8ab4563ca804ef1ba2c5a6d8e0647a72cba9d3584c4ecfaf"}
Oct 30 03:15:02.956013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.955990    4278 generic.go:356] "Generic (PLEG): container finished" podID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerID="79549b234801ab6e746f20e20252f8f817f82b276e960f6f033ee72320866a26" exitCode=60
Oct 30 03:15:02.956124 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.956094    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws" event={"ID":"9e75f3b5-4d25-4fed-91f1-99a41800f320","Type":"ContainerDied","Data":"79549b234801ab6e746f20e20252f8f817f82b276e960f6f033ee72320866a26"}
Oct 30 03:15:02.956385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.956364    4278 scope.go:122] "RemoveContainer" containerID="79549b234801ab6e746f20e20252f8f817f82b276e960f6f033ee72320866a26"
Oct 30 03:15:02.958006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.957985    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" event={"ID":"8e6f2b85-9672-4fc7-8a8a-0393e7886187","Type":"ContainerStarted","Data":"2eb2228afe928db3af1ff1eb4d15bed65e7533b206a42985d9e59aa2302b9f8b"}
Oct 30 03:15:02.978901 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:02.978825    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" podStartSLOduration=2.978805432 podStartE2EDuration="2.978805432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:02.973082583 +0000 UTC m=+2040.897290772" watchObservedRunningTime="2026-10-30 03:15:02.978805432 +0000 UTC m=+2040.903013621"
Oct 30 03:15:03.015269 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.015218    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" podStartSLOduration=3.015202083 podStartE2EDuration="3.015202083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.012194075 +0000 UTC m=+2040.936402286" watchObservedRunningTime="2026-10-30 03:15:03.015202083 +0000 UTC m=+2040.939410262"
Oct 30 03:15:03.051185 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.051090    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" podStartSLOduration=3.051063768 podStartE2EDuration="3.051063768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.050044791 +0000 UTC m=+2040.974252979" watchObservedRunningTime="2026-10-30 03:15:03.051063768 +0000 UTC m=+2040.975271957"
Oct 30 03:15:03.179557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.179500    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" podStartSLOduration=3.179481343 podStartE2EDuration="3.179481343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.177567897 +0000 UTC m=+2041.101776085" watchObservedRunningTime="2026-10-30 03:15:03.179481343 +0000 UTC m=+2041.103689529"
Oct 30 03:15:03.220763 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.220636    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" podStartSLOduration=3.220621364 podStartE2EDuration="3.220621364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.219257149 +0000 UTC m=+2041.143465337" watchObservedRunningTime="2026-10-30 03:15:03.220621364 +0000 UTC m=+2041.144829552"
Oct 30 03:15:03.264505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.264161    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" podStartSLOduration=3.264145338 podStartE2EDuration="3.264145338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.25874763 +0000 UTC m=+2041.182955829" watchObservedRunningTime="2026-10-30 03:15:03.264145338 +0000 UTC m=+2041.188353581"
Oct 30 03:15:03.346670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.346599    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" podStartSLOduration=3.346578593 podStartE2EDuration="3.346578593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.339955551 +0000 UTC m=+2041.264163738" watchObservedRunningTime="2026-10-30 03:15:03.346578593 +0000 UTC m=+2041.270786784"
Oct 30 03:15:03.512614 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.512577    4278 scope.go:122] "RemoveContainer" containerID="40baaef43c41bb248d90933775bc5c95af8aa75b3d6996c58c048bc3bc5ad2e7"
Oct 30 03:15:03.713516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.713290    4278 scope.go:122] "RemoveContainer" containerID="478472dca9a9ad1c5eb6b3267d50ed4e3d1ac7af3e989e2e8c400de2044254e3"
Oct 30 03:15:03.732171 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.732148    4278 scope.go:122] "RemoveContainer" containerID="a8f621eb6c2e45b5b34c03a7a4d9bbaf1e39ab2f320fb491af4be69e22179f3e"
Oct 30 03:15:03.975043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.974073    4278 generic.go:356] "Generic (PLEG): container finished" podID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerID="b5c315e80fe884ef34046479878b5b99ab00aaf94028e2068afcffde4ac96e46" exitCode=60
Oct 30 03:15:03.975043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.974156    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" event={"ID":"fed730a6-41b7-4cfc-af55-ae03dbc48d38","Type":"ContainerDied","Data":"b5c315e80fe884ef34046479878b5b99ab00aaf94028e2068afcffde4ac96e46"}
Oct 30 03:15:03.975043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.974637    4278 scope.go:122] "RemoveContainer" containerID="b5c315e80fe884ef34046479878b5b99ab00aaf94028e2068afcffde4ac96e46"
Oct 30 03:15:03.977488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.977401    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6" event={"ID":"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00","Type":"ContainerStarted","Data":"7aa1fa386c00498797d879f8b9a063fdbd53968b061c7ea9d993d30e382e3379"}
Oct 30 03:15:03.984810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.984785    4278 generic.go:356] "Generic (PLEG): container finished" podID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerID="bd547fbae002c94c379ce4443b8fd1568ac512afe2faf0645c6fbb6eeb0d4e4c" exitCode=60
Oct 30 03:15:03.984954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.984875    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" event={"ID":"969c6501-88c9-4285-9b60-2d38d788bbd6","Type":"ContainerDied","Data":"bd547fbae002c94c379ce4443b8fd1568ac512afe2faf0645c6fbb6eeb0d4e4c"}
Oct 30 03:15:03.985339 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.985298    4278 scope.go:122] "RemoveContainer" containerID="bd547fbae002c94c379ce4443b8fd1568ac512afe2faf0645c6fbb6eeb0d4e4c"
Oct 30 03:15:03.989186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.989165    4278 generic.go:356] "Generic (PLEG): container finished" podID="ea6c297a-e973-4e59-859f-5c4862900275" containerID="e2ab0fd0b596fe7f494693f16bea01a11ab65789fb585b0079414f00084e6de2" exitCode=60
Oct 30 03:15:03.989288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.989242    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" event={"ID":"ea6c297a-e973-4e59-859f-5c4862900275","Type":"ContainerDied","Data":"e2ab0fd0b596fe7f494693f16bea01a11ab65789fb585b0079414f00084e6de2"}
Oct 30 03:15:03.989584 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.989563    4278 scope.go:122] "RemoveContainer" containerID="e2ab0fd0b596fe7f494693f16bea01a11ab65789fb585b0079414f00084e6de2"
Oct 30 03:15:03.993852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.993709    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" event={"ID":"c5968a2a-86ba-4717-8f0f-df111b79f123","Type":"ContainerStarted","Data":"5eaf7d94586e5a5cbc224674a79529cf6875cc405975ac94a913fe0641be0e85"}
Oct 30 03:15:03.995562 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.995537    4278 generic.go:356] "Generic (PLEG): container finished" podID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerID="def5c559f06cef1b961e32ebd390d00300599a548928bb02fecfbfd24e06e299" exitCode=60
Oct 30 03:15:03.995657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.995618    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" event={"ID":"8223ce14-326f-40bd-bb96-dc1b24bb4fd0","Type":"ContainerDied","Data":"def5c559f06cef1b961e32ebd390d00300599a548928bb02fecfbfd24e06e299"}
Oct 30 03:15:03.995884 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.995864    4278 scope.go:122] "RemoveContainer" containerID="def5c559f06cef1b961e32ebd390d00300599a548928bb02fecfbfd24e06e299"
Oct 30 03:15:03.997598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.997576    4278 generic.go:356] "Generic (PLEG): container finished" podID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerID="c66da6417ba6604f6b3b1b5a089d4a32510786bb5a56c04849889b553eb51d31" exitCode=60
Oct 30 03:15:03.997704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.997602    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" event={"ID":"5103b31a-7eaf-4364-9af6-1b7eb34e1d40","Type":"ContainerDied","Data":"c66da6417ba6604f6b3b1b5a089d4a32510786bb5a56c04849889b553eb51d31"}
Oct 30 03:15:03.997992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.997958    4278 scope.go:122] "RemoveContainer" containerID="c66da6417ba6604f6b3b1b5a089d4a32510786bb5a56c04849889b553eb51d31"
Oct 30 03:15:03.999738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.999712    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" event={"ID":"0d2f4868-04fa-430b-93bc-9644c7a526cd","Type":"ContainerStarted","Data":"148dfb1f05c7f2dc264cccfdac8da6e6a39d944507066c752a82f7037a8f4c01"}
Oct 30 03:15:04.000017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:03.999996    4278 scope.go:122] "RemoveContainer" containerID="148dfb1f05c7f2dc264cccfdac8da6e6a39d944507066c752a82f7037a8f4c01"
Oct 30 03:15:04.001772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.001476    4278 generic.go:356] "Generic (PLEG): container finished" podID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerID="37e4b24bb2a2856492f6af8474804fb6065591a5c17b86c5ffa7c70600d84601" exitCode=60
Oct 30 03:15:04.001772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.001649    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7" event={"ID":"a02df413-cc1b-4d7e-8209-16022147c7a1","Type":"ContainerDied","Data":"37e4b24bb2a2856492f6af8474804fb6065591a5c17b86c5ffa7c70600d84601"}
Oct 30 03:15:04.002046 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.001783    4278 scope.go:122] "RemoveContainer" containerID="37e4b24bb2a2856492f6af8474804fb6065591a5c17b86c5ffa7c70600d84601"
Oct 30 03:15:04.023664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.023627    4278 generic.go:356] "Generic (PLEG): container finished" podID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerID="818eb5259fed8daa8ab4563ca804ef1ba2c5a6d8e0647a72cba9d3584c4ecfaf" exitCode=60
Oct 30 03:15:04.023836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.023813    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" event={"ID":"6c90ec76-2fe7-40e2-b88b-8b766b28e158","Type":"ContainerDied","Data":"818eb5259fed8daa8ab4563ca804ef1ba2c5a6d8e0647a72cba9d3584c4ecfaf"}
Oct 30 03:15:04.025219 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.024470    4278 scope.go:122] "RemoveContainer" containerID="818eb5259fed8daa8ab4563ca804ef1ba2c5a6d8e0647a72cba9d3584c4ecfaf"
Oct 30 03:15:04.028289 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.027081    4278 generic.go:356] "Generic (PLEG): container finished" podID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerID="2eb2228afe928db3af1ff1eb4d15bed65e7533b206a42985d9e59aa2302b9f8b" exitCode=60
Oct 30 03:15:04.028399 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.028309    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" event={"ID":"8e6f2b85-9672-4fc7-8a8a-0393e7886187","Type":"ContainerDied","Data":"2eb2228afe928db3af1ff1eb4d15bed65e7533b206a42985d9e59aa2302b9f8b"}
Oct 30 03:15:04.029397 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.028473    4278 scope.go:122] "RemoveContainer" containerID="2eb2228afe928db3af1ff1eb4d15bed65e7533b206a42985d9e59aa2302b9f8b"
Oct 30 03:15:04.215238 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.215180    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6" podStartSLOduration=4.215163854 podStartE2EDuration="4.215163854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:04.213656293 +0000 UTC m=+2042.137864482" watchObservedRunningTime="2026-10-30 03:15:04.215163854 +0000 UTC m=+2042.139372042"
Oct 30 03:15:04.252645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.252580    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" podStartSLOduration=4.252563082 podStartE2EDuration="4.252563082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:04.252263352 +0000 UTC m=+2042.176471541" watchObservedRunningTime="2026-10-30 03:15:04.252563082 +0000 UTC m=+2042.176771270"
Oct 30 03:15:04.378124 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:04.378058    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx" podStartSLOduration=4.378039808 podStartE2EDuration="4.378039808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:04.376244214 +0000 UTC m=+2042.300452404" watchObservedRunningTime="2026-10-30 03:15:04.378039808 +0000 UTC m=+2042.302247998"
Oct 30 03:15:05.038541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.038498    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7" event={"ID":"a02df413-cc1b-4d7e-8209-16022147c7a1","Type":"ContainerStarted","Data":"c391b6e1031c1c70f65e0016b6eccf77b0f5e13e5428356584ace6503021a818"}
Oct 30 03:15:05.041053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.041015    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" event={"ID":"6c90ec76-2fe7-40e2-b88b-8b766b28e158","Type":"ContainerStarted","Data":"9464e3e3200b0d01f5dc529645fdac2137a4a2a8f5ba796336f6fe97f077a9a1"}
Oct 30 03:15:05.043140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.043110    4278 generic.go:356] "Generic (PLEG): container finished" podID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerID="81e9f74f8253e36dc415be57ccda6dd9f848176b5a24c2d4d57bdbd44066713f" exitCode=60
Oct 30 03:15:05.043296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.043190    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws" event={"ID":"9e75f3b5-4d25-4fed-91f1-99a41800f320","Type":"ContainerDied","Data":"81e9f74f8253e36dc415be57ccda6dd9f848176b5a24c2d4d57bdbd44066713f"}
Oct 30 03:15:05.043296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.043254    4278 scope.go:122] "RemoveContainer" containerID="79549b234801ab6e746f20e20252f8f817f82b276e960f6f033ee72320866a26"
Oct 30 03:15:05.043529 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.043510    4278 scope.go:122] "RemoveContainer" containerID="81e9f74f8253e36dc415be57ccda6dd9f848176b5a24c2d4d57bdbd44066713f"
Oct 30 03:15:05.043911 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:05.043878    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws_odh-ai-gateway-infra(9e75f3b5-4d25-4fed-91f1-99a41800f320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320"
Oct 30 03:15:05.045636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.045613    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" event={"ID":"8e6f2b85-9672-4fc7-8a8a-0393e7886187","Type":"ContainerStarted","Data":"fb0f4445b5266656f37cc38ea603f6c310d09cee7606753905425027417ae559"}
Oct 30 03:15:05.047643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.047617    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" event={"ID":"fed730a6-41b7-4cfc-af55-ae03dbc48d38","Type":"ContainerStarted","Data":"cade8d0f2a698961c9697ffed7a94c8f89b69c01a67c63adff55b66ea358927d"}
Oct 30 03:15:05.049810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.049782    4278 generic.go:356] "Generic (PLEG): container finished" podID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerID="7aa1fa386c00498797d879f8b9a063fdbd53968b061c7ea9d993d30e382e3379" exitCode=60
Oct 30 03:15:05.049934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.049880    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6" event={"ID":"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00","Type":"ContainerDied","Data":"7aa1fa386c00498797d879f8b9a063fdbd53968b061c7ea9d993d30e382e3379"}
Oct 30 03:15:05.050101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.050083    4278 scope.go:122] "RemoveContainer" containerID="7aa1fa386c00498797d879f8b9a063fdbd53968b061c7ea9d993d30e382e3379"
Oct 30 03:15:05.050468 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:05.050416    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6_odh-ai-gateway-infra(4a1d34d8-4d62-4a52-bb97-ae9ea24eac00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00"
Oct 30 03:15:05.052181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.052157    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" event={"ID":"969c6501-88c9-4285-9b60-2d38d788bbd6","Type":"ContainerStarted","Data":"023932c4073a760e749b8c42efc246231562304dac1087ca36f281a3128d1975"}
Oct 30 03:15:05.054313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.054292    4278 generic.go:356] "Generic (PLEG): container finished" podID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerID="9535cd2bb9a993e773b973b31bd4ce8ee741f1563eafa83676c2a9b978b60406" exitCode=60
Oct 30 03:15:05.054399 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.054366    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx" event={"ID":"bfefef3e-400b-48d3-b4f8-bf432ee80c29","Type":"ContainerDied","Data":"9535cd2bb9a993e773b973b31bd4ce8ee741f1563eafa83676c2a9b978b60406"}
Oct 30 03:15:05.054772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.054756    4278 scope.go:122] "RemoveContainer" containerID="9535cd2bb9a993e773b973b31bd4ce8ee741f1563eafa83676c2a9b978b60406"
Oct 30 03:15:05.055087 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:05.055059    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx_odh-ai-gateway-infra(bfefef3e-400b-48d3-b4f8-bf432ee80c29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29"
Oct 30 03:15:05.056701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.056676    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" event={"ID":"ea6c297a-e973-4e59-859f-5c4862900275","Type":"ContainerStarted","Data":"aa669a9b04391434c65fec795ff96e1a3bd5252acb7d7677b7f1d17d9704068b"}
Oct 30 03:15:05.059951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.059921    4278 generic.go:356] "Generic (PLEG): container finished" podID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerID="5eaf7d94586e5a5cbc224674a79529cf6875cc405975ac94a913fe0641be0e85" exitCode=60
Oct 30 03:15:05.060061 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.060021    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" event={"ID":"c5968a2a-86ba-4717-8f0f-df111b79f123","Type":"ContainerDied","Data":"5eaf7d94586e5a5cbc224674a79529cf6875cc405975ac94a913fe0641be0e85"}
Oct 30 03:15:05.060468 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.060393    4278 scope.go:122] "RemoveContainer" containerID="5eaf7d94586e5a5cbc224674a79529cf6875cc405975ac94a913fe0641be0e85"
Oct 30 03:15:05.060770 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:05.060744    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf_odh-ai-gateway-infra(c5968a2a-86ba-4717-8f0f-df111b79f123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123"
Oct 30 03:15:05.060897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.060854    4278 scope.go:122] "RemoveContainer" containerID="cfffb6ecd772ef1143711ddc1645ba0c7b005bb6f0dc05116e6ce56f151b73a8"
Oct 30 03:15:05.066895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.066865    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" event={"ID":"8223ce14-326f-40bd-bb96-dc1b24bb4fd0","Type":"ContainerStarted","Data":"ccf09e796dd3f7dd9d06ccfd92bbfdefca1ed2d16a7e37b47cfefd9524437f5f"}
Oct 30 03:15:05.069172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.069143    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" event={"ID":"5103b31a-7eaf-4364-9af6-1b7eb34e1d40","Type":"ContainerStarted","Data":"953d6fa9016778fe8e26d02dcb90418a2311237a71d189ce1da8aa54a22794b1"}
Oct 30 03:15:05.071345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.071316    4278 generic.go:356] "Generic (PLEG): container finished" podID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerID="148dfb1f05c7f2dc264cccfdac8da6e6a39d944507066c752a82f7037a8f4c01" exitCode=60
Oct 30 03:15:05.071508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.071362    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" event={"ID":"0d2f4868-04fa-430b-93bc-9644c7a526cd","Type":"ContainerDied","Data":"148dfb1f05c7f2dc264cccfdac8da6e6a39d944507066c752a82f7037a8f4c01"}
Oct 30 03:15:05.071508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.071404    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" event={"ID":"0d2f4868-04fa-430b-93bc-9644c7a526cd","Type":"ContainerStarted","Data":"94d9d9e4f21376005dfccc935467fd73d440bc82f8c11d30814f779f6a88a0fd"}
Oct 30 03:15:05.078847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.078820    4278 scope.go:122] "RemoveContainer" containerID="244e5c0b29d8e4b772a43c10b6a4599e5a4f7fa5f9aa4b224a9d6efa182d1496"
Oct 30 03:15:05.084901 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.084847    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7" podStartSLOduration=5.084831648 podStartE2EDuration="5.084831648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:05.081015928 +0000 UTC m=+2043.005224116" watchObservedRunningTime="2026-10-30 03:15:05.084831648 +0000 UTC m=+2043.009039835"
Oct 30 03:15:05.098012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.097983    4278 scope.go:122] "RemoveContainer" containerID="07ab3ff6de4e6d77a3b2f561aed388cf2901f563d5b414437de891085b6830e0"
Oct 30 03:15:05.289678 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:05.289616    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" podStartSLOduration=5.289598797 podStartE2EDuration="5.289598797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:05.286093065 +0000 UTC m=+2043.210301253" watchObservedRunningTime="2026-10-30 03:15:05.289598797 +0000 UTC m=+2043.213806984"
Oct 30 03:15:06.099164 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.099126    4278 generic.go:356] "Generic (PLEG): container finished" podID="ea6c297a-e973-4e59-859f-5c4862900275" containerID="aa669a9b04391434c65fec795ff96e1a3bd5252acb7d7677b7f1d17d9704068b" exitCode=60
Oct 30 03:15:06.099672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.099408    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" event={"ID":"ea6c297a-e973-4e59-859f-5c4862900275","Type":"ContainerDied","Data":"aa669a9b04391434c65fec795ff96e1a3bd5252acb7d7677b7f1d17d9704068b"}
Oct 30 03:15:06.099672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.099661    4278 scope.go:122] "RemoveContainer" containerID="e2ab0fd0b596fe7f494693f16bea01a11ab65789fb585b0079414f00084e6de2"
Oct 30 03:15:06.099765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.099731    4278 scope.go:122] "RemoveContainer" containerID="aa669a9b04391434c65fec795ff96e1a3bd5252acb7d7677b7f1d17d9704068b"
Oct 30 03:15:06.100255 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.100225    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x_odh-ai-gateway-infra(ea6c297a-e973-4e59-859f-5c4862900275)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" podUID="ea6c297a-e973-4e59-859f-5c4862900275"
Oct 30 03:15:06.102099 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.102073    4278 scope.go:122] "RemoveContainer" containerID="5eaf7d94586e5a5cbc224674a79529cf6875cc405975ac94a913fe0641be0e85"
Oct 30 03:15:06.102573 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.102544    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf_odh-ai-gateway-infra(c5968a2a-86ba-4717-8f0f-df111b79f123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123"
Oct 30 03:15:06.105153 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.105124    4278 generic.go:356] "Generic (PLEG): container finished" podID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerID="ccf09e796dd3f7dd9d06ccfd92bbfdefca1ed2d16a7e37b47cfefd9524437f5f" exitCode=60
Oct 30 03:15:06.105276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.105159    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" event={"ID":"8223ce14-326f-40bd-bb96-dc1b24bb4fd0","Type":"ContainerDied","Data":"ccf09e796dd3f7dd9d06ccfd92bbfdefca1ed2d16a7e37b47cfefd9524437f5f"}
Oct 30 03:15:06.106275 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.106138    4278 scope.go:122] "RemoveContainer" containerID="ccf09e796dd3f7dd9d06ccfd92bbfdefca1ed2d16a7e37b47cfefd9524437f5f"
Oct 30 03:15:06.108385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.108358    4278 generic.go:356] "Generic (PLEG): container finished" podID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerID="953d6fa9016778fe8e26d02dcb90418a2311237a71d189ce1da8aa54a22794b1" exitCode=60
Oct 30 03:15:06.108539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.108452    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" event={"ID":"5103b31a-7eaf-4364-9af6-1b7eb34e1d40","Type":"ContainerDied","Data":"953d6fa9016778fe8e26d02dcb90418a2311237a71d189ce1da8aa54a22794b1"}
Oct 30 03:15:06.108931 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.108912    4278 scope.go:122] "RemoveContainer" containerID="953d6fa9016778fe8e26d02dcb90418a2311237a71d189ce1da8aa54a22794b1"
Oct 30 03:15:06.109022 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.109005    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj_odh-ai-gateway-infra(8223ce14-326f-40bd-bb96-dc1b24bb4fd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0"
Oct 30 03:15:06.109307 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.109283    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j_odh-ai-gateway-infra(5103b31a-7eaf-4364-9af6-1b7eb34e1d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40"
Oct 30 03:15:06.112210 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.112182    4278 generic.go:356] "Generic (PLEG): container finished" podID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerID="94d9d9e4f21376005dfccc935467fd73d440bc82f8c11d30814f779f6a88a0fd" exitCode=60
Oct 30 03:15:06.112693 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.112272    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" event={"ID":"0d2f4868-04fa-430b-93bc-9644c7a526cd","Type":"ContainerDied","Data":"94d9d9e4f21376005dfccc935467fd73d440bc82f8c11d30814f779f6a88a0fd"}
Oct 30 03:15:06.112815 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.112611    4278 scope.go:122] "RemoveContainer" containerID="94d9d9e4f21376005dfccc935467fd73d440bc82f8c11d30814f779f6a88a0fd"
Oct 30 03:15:06.113243 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.113216    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb_odh-ai-gateway-infra(0d2f4868-04fa-430b-93bc-9644c7a526cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd"
Oct 30 03:15:06.151317 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.150829    4278 scope.go:122] "RemoveContainer" containerID="def5c559f06cef1b961e32ebd390d00300599a548928bb02fecfbfd24e06e299"
Oct 30 03:15:06.151317 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.151074    4278 generic.go:356] "Generic (PLEG): container finished" podID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerID="c391b6e1031c1c70f65e0016b6eccf77b0f5e13e5428356584ace6503021a818" exitCode=60
Oct 30 03:15:06.151317 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.151172    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7" event={"ID":"a02df413-cc1b-4d7e-8209-16022147c7a1","Type":"ContainerDied","Data":"c391b6e1031c1c70f65e0016b6eccf77b0f5e13e5428356584ace6503021a818"}
Oct 30 03:15:06.154077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.154050    4278 scope.go:122] "RemoveContainer" containerID="c391b6e1031c1c70f65e0016b6eccf77b0f5e13e5428356584ace6503021a818"
Oct 30 03:15:06.154602 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.154570    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7_odh-ai-gateway-infra(a02df413-cc1b-4d7e-8209-16022147c7a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1"
Oct 30 03:15:06.157190 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.157162    4278 generic.go:356] "Generic (PLEG): container finished" podID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerID="9464e3e3200b0d01f5dc529645fdac2137a4a2a8f5ba796336f6fe97f077a9a1" exitCode=60
Oct 30 03:15:06.157304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.157245    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" event={"ID":"6c90ec76-2fe7-40e2-b88b-8b766b28e158","Type":"ContainerDied","Data":"9464e3e3200b0d01f5dc529645fdac2137a4a2a8f5ba796336f6fe97f077a9a1"}
Oct 30 03:15:06.157724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.157701    4278 scope.go:122] "RemoveContainer" containerID="9464e3e3200b0d01f5dc529645fdac2137a4a2a8f5ba796336f6fe97f077a9a1"
Oct 30 03:15:06.158179 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.158152    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc_odh-ai-gateway-infra(6c90ec76-2fe7-40e2-b88b-8b766b28e158)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158"
Oct 30 03:15:06.159845 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.159822    4278 scope.go:122] "RemoveContainer" containerID="81e9f74f8253e36dc415be57ccda6dd9f848176b5a24c2d4d57bdbd44066713f"
Oct 30 03:15:06.160271 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.160245    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws_odh-ai-gateway-infra(9e75f3b5-4d25-4fed-91f1-99a41800f320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320"
Oct 30 03:15:06.162367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.162347    4278 generic.go:356] "Generic (PLEG): container finished" podID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerID="fb0f4445b5266656f37cc38ea603f6c310d09cee7606753905425027417ae559" exitCode=60
Oct 30 03:15:06.162569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.162547    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" event={"ID":"8e6f2b85-9672-4fc7-8a8a-0393e7886187","Type":"ContainerDied","Data":"fb0f4445b5266656f37cc38ea603f6c310d09cee7606753905425027417ae559"}
Oct 30 03:15:06.162878 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.162860    4278 scope.go:122] "RemoveContainer" containerID="fb0f4445b5266656f37cc38ea603f6c310d09cee7606753905425027417ae559"
Oct 30 03:15:06.163232 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.163213    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb_odh-ai-gateway-infra(8e6f2b85-9672-4fc7-8a8a-0393e7886187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187"
Oct 30 03:15:06.164793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.164769    4278 generic.go:356] "Generic (PLEG): container finished" podID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerID="cade8d0f2a698961c9697ffed7a94c8f89b69c01a67c63adff55b66ea358927d" exitCode=60
Oct 30 03:15:06.164896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.164847    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" event={"ID":"fed730a6-41b7-4cfc-af55-ae03dbc48d38","Type":"ContainerDied","Data":"cade8d0f2a698961c9697ffed7a94c8f89b69c01a67c63adff55b66ea358927d"}
Oct 30 03:15:06.165129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.165111    4278 scope.go:122] "RemoveContainer" containerID="cade8d0f2a698961c9697ffed7a94c8f89b69c01a67c63adff55b66ea358927d"
Oct 30 03:15:06.165569 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.165545    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws_odh-ai-gateway-infra(fed730a6-41b7-4cfc-af55-ae03dbc48d38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38"
Oct 30 03:15:06.167124 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.167100    4278 scope.go:122] "RemoveContainer" containerID="7aa1fa386c00498797d879f8b9a063fdbd53968b061c7ea9d993d30e382e3379"
Oct 30 03:15:06.167512 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.167413    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6_odh-ai-gateway-infra(4a1d34d8-4d62-4a52-bb97-ae9ea24eac00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00"
Oct 30 03:15:06.168971 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.168945    4278 generic.go:356] "Generic (PLEG): container finished" podID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerID="023932c4073a760e749b8c42efc246231562304dac1087ca36f281a3128d1975" exitCode=60
Oct 30 03:15:06.169062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.168987    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" event={"ID":"969c6501-88c9-4285-9b60-2d38d788bbd6","Type":"ContainerDied","Data":"023932c4073a760e749b8c42efc246231562304dac1087ca36f281a3128d1975"}
Oct 30 03:15:06.169251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.169238    4278 scope.go:122] "RemoveContainer" containerID="023932c4073a760e749b8c42efc246231562304dac1087ca36f281a3128d1975"
Oct 30 03:15:06.169549 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:06.169530    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4_odh-ai-gateway-infra(969c6501-88c9-4285-9b60-2d38d788bbd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6"
Oct 30 03:15:06.173028 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.173005    4278 scope.go:122] "RemoveContainer" containerID="c66da6417ba6604f6b3b1b5a089d4a32510786bb5a56c04849889b553eb51d31"
Oct 30 03:15:06.189308 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.189275    4278 scope.go:122] "RemoveContainer" containerID="148dfb1f05c7f2dc264cccfdac8da6e6a39d944507066c752a82f7037a8f4c01"
Oct 30 03:15:06.206810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.206787    4278 scope.go:122] "RemoveContainer" containerID="37e4b24bb2a2856492f6af8474804fb6065591a5c17b86c5ffa7c70600d84601"
Oct 30 03:15:06.222091 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.222069    4278 scope.go:122] "RemoveContainer" containerID="818eb5259fed8daa8ab4563ca804ef1ba2c5a6d8e0647a72cba9d3584c4ecfaf"
Oct 30 03:15:06.238679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.238652    4278 scope.go:122] "RemoveContainer" containerID="2eb2228afe928db3af1ff1eb4d15bed65e7533b206a42985d9e59aa2302b9f8b"
Oct 30 03:15:06.256724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.256694    4278 scope.go:122] "RemoveContainer" containerID="b5c315e80fe884ef34046479878b5b99ab00aaf94028e2068afcffde4ac96e46"
Oct 30 03:15:06.279948 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:06.279920    4278 scope.go:122] "RemoveContainer" containerID="bd547fbae002c94c379ce4443b8fd1568ac512afe2faf0645c6fbb6eeb0d4e4c"
Oct 30 03:15:07.178788 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:07.178756    4278 scope.go:122] "RemoveContainer" containerID="9464e3e3200b0d01f5dc529645fdac2137a4a2a8f5ba796336f6fe97f077a9a1"
Oct 30 03:15:07.179263 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:07.179235    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc_odh-ai-gateway-infra(6c90ec76-2fe7-40e2-b88b-8b766b28e158)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158"
Oct 30 03:15:07.180689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:07.180662    4278 scope.go:122] "RemoveContainer" containerID="fb0f4445b5266656f37cc38ea603f6c310d09cee7606753905425027417ae559"
Oct 30 03:15:07.181059 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:07.181037    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb_odh-ai-gateway-infra(8e6f2b85-9672-4fc7-8a8a-0393e7886187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187"
Oct 30 03:15:07.182457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:07.182406    4278 scope.go:122] "RemoveContainer" containerID="cade8d0f2a698961c9697ffed7a94c8f89b69c01a67c63adff55b66ea358927d"
Oct 30 03:15:07.182812 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:07.182788    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws_odh-ai-gateway-infra(fed730a6-41b7-4cfc-af55-ae03dbc48d38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38"
Oct 30 03:15:07.184177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:07.184159    4278 scope.go:122] "RemoveContainer" containerID="023932c4073a760e749b8c42efc246231562304dac1087ca36f281a3128d1975"
Oct 30 03:15:07.184546 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:07.184525    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4_odh-ai-gateway-infra(969c6501-88c9-4285-9b60-2d38d788bbd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6"
Oct 30 03:15:07.185922 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:07.185905    4278 scope.go:122] "RemoveContainer" containerID="aa669a9b04391434c65fec795ff96e1a3bd5252acb7d7677b7f1d17d9704068b"
Oct 30 03:15:07.186270 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:07.186243    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x_odh-ai-gateway-infra(ea6c297a-e973-4e59-859f-5c4862900275)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" podUID="ea6c297a-e973-4e59-859f-5c4862900275"
Oct 30 03:15:07.187642 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:07.187621    4278 scope.go:122] "RemoveContainer" containerID="ccf09e796dd3f7dd9d06ccfd92bbfdefca1ed2d16a7e37b47cfefd9524437f5f"
Oct 30 03:15:07.187989 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:07.187958    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj_odh-ai-gateway-infra(8223ce14-326f-40bd-bb96-dc1b24bb4fd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0"
Oct 30 03:15:07.189460 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:07.189439    4278 scope.go:122] "RemoveContainer" containerID="953d6fa9016778fe8e26d02dcb90418a2311237a71d189ce1da8aa54a22794b1"
Oct 30 03:15:07.189766 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:07.189744    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j_odh-ai-gateway-infra(5103b31a-7eaf-4364-9af6-1b7eb34e1d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40"
Oct 30 03:15:07.191143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:07.191122    4278 scope.go:122] "RemoveContainer" containerID="94d9d9e4f21376005dfccc935467fd73d440bc82f8c11d30814f779f6a88a0fd"
Oct 30 03:15:07.191459 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:07.191403    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb_odh-ai-gateway-infra(0d2f4868-04fa-430b-93bc-9644c7a526cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd"
Oct 30 03:15:07.192864 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:07.192844    4278 scope.go:122] "RemoveContainer" containerID="c391b6e1031c1c70f65e0016b6eccf77b0f5e13e5428356584ace6503021a818"
Oct 30 03:15:07.193105 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:07.193084    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7_odh-ai-gateway-infra(a02df413-cc1b-4d7e-8209-16022147c7a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1"
Oct 30 03:15:10.384195 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:10.384146    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:10.397948 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:10.397919    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:11.243110 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:11.243076    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:15:15.012624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.011786    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"]
Oct 30 03:15:15.028104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.028066    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.038889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.038348    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"]
Oct 30 03:15:15.061200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.061158    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f21058\"/\"test-model-f21058-kserve-self-signed-certs\""
Oct 30 03:15:15.061369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.061341    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f21058\"/\"kube-root-ca.crt\""
Oct 30 03:15:15.061524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.061466    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f21058\"/\"default-dockercfg-gx4r6\""
Oct 30 03:15:15.061632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.061599    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f21058\"/\"openshift-service-ca.crt\""
Oct 30 03:15:15.163552 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.163515    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-model-cache\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.163747 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.163592    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dskst\" (UniqueName: \"kubernetes.io/projected/b412a03c-6675-4269-a2e0-b0cd5a059d39-kube-api-access-dskst\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.163747 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.163722    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-tmp-dir\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.163884 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.163865    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b412a03c-6675-4269-a2e0-b0cd5a059d39-tls-certs\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.163947 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.163930    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-dshm\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.164051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.164035    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-home\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.192579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.192548    4278 scope.go:122] "RemoveContainer" containerID="5eaf7d94586e5a5cbc224674a79529cf6875cc405975ac94a913fe0641be0e85"
Oct 30 03:15:15.267528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.266940    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-model-cache\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.267528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.267041    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskst\" (UniqueName: \"kubernetes.io/projected/b412a03c-6675-4269-a2e0-b0cd5a059d39-kube-api-access-dskst\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.267528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.267081    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-tmp-dir\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.267528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.267180    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b412a03c-6675-4269-a2e0-b0cd5a059d39-tls-certs\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.267528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.267227    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-dshm\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.267528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.267303    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-home\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.278456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.268418    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-home\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.278456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.268782    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-model-cache\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.278873 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.278845    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-tmp-dir\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.301483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.299148    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dskst\" (UniqueName: \"kubernetes.io/projected/b412a03c-6675-4269-a2e0-b0cd5a059d39-kube-api-access-dskst\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.356447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.345782    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-dshm\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.387517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.387480    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b412a03c-6675-4269-a2e0-b0cd5a059d39-tls-certs\") pod \"test-model-f21058-kserve-6cbc669967-fm22r\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") " pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:15.650259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:15.650214    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:16.014463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:16.014411    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"]
Oct 30 03:15:16.014813 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:15:16.014656    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb412a03c_6675_4269_a2e0_b0cd5a059d39.slice/crio-46835f5bf52578bf0edee080a61832fe5263776fed25dd516a1f8152faf34690 WatchSource:0}: Error finding container 46835f5bf52578bf0edee080a61832fe5263776fed25dd516a1f8152faf34690: Status 404 returned error can't find the container with id 46835f5bf52578bf0edee080a61832fe5263776fed25dd516a1f8152faf34690
Oct 30 03:15:16.192838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:16.192792    4278 scope.go:122] "RemoveContainer" containerID="7aa1fa386c00498797d879f8b9a063fdbd53968b061c7ea9d993d30e382e3379"
Oct 30 03:15:16.193038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:16.192939    4278 scope.go:122] "RemoveContainer" containerID="023932c4073a760e749b8c42efc246231562304dac1087ca36f281a3128d1975"
Oct 30 03:15:16.402644 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:16.402520    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" event={"ID":"c5968a2a-86ba-4717-8f0f-df111b79f123","Type":"ContainerStarted","Data":"90714a00bf9feba3fd30110b1a58faaa3ca71d228e3f6739af7e10d1480d3479"}
Oct 30 03:15:16.404986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:16.404952    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r" event={"ID":"b412a03c-6675-4269-a2e0-b0cd5a059d39","Type":"ContainerStarted","Data":"88d31c31d1096dd781640f542b6cf1817c9eb4c18440d00b9bccaf6673ca7a8f"}
Oct 30 03:15:16.405179 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:16.405000    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r" event={"ID":"b412a03c-6675-4269-a2e0-b0cd5a059d39","Type":"ContainerStarted","Data":"46835f5bf52578bf0edee080a61832fe5263776fed25dd516a1f8152faf34690"}
Oct 30 03:15:16.490667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:16.490604    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r" podStartSLOduration=2.490583741 podStartE2EDuration="2.490583741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:16.484687534 +0000 UTC m=+2054.408895722" watchObservedRunningTime="2026-10-30 03:15:16.490583741 +0000 UTC m=+2054.414791933"
Oct 30 03:15:16.650539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:16.650496    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:17.192054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.192020    4278 scope.go:122] "RemoveContainer" containerID="9535cd2bb9a993e773b973b31bd4ce8ee741f1563eafa83676c2a9b978b60406"
Oct 30 03:15:17.192589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.192565    4278 scope.go:122] "RemoveContainer" containerID="c391b6e1031c1c70f65e0016b6eccf77b0f5e13e5428356584ace6503021a818"
Oct 30 03:15:17.192921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.192822    4278 scope.go:122] "RemoveContainer" containerID="9464e3e3200b0d01f5dc529645fdac2137a4a2a8f5ba796336f6fe97f077a9a1"
Oct 30 03:15:17.415769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.415676    4278 generic.go:356] "Generic (PLEG): container finished" podID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerID="e0b43592cab04e54b17ca90ae4405087764cc4f727873b633a5b1fb99bf8b169" exitCode=60
Oct 30 03:15:17.415769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.415748    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6" event={"ID":"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00","Type":"ContainerDied","Data":"e0b43592cab04e54b17ca90ae4405087764cc4f727873b633a5b1fb99bf8b169"}
Oct 30 03:15:17.416005 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.415807    4278 scope.go:122] "RemoveContainer" containerID="7aa1fa386c00498797d879f8b9a063fdbd53968b061c7ea9d993d30e382e3379"
Oct 30 03:15:17.416344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.416317    4278 scope.go:122] "RemoveContainer" containerID="e0b43592cab04e54b17ca90ae4405087764cc4f727873b633a5b1fb99bf8b169"
Oct 30 03:15:17.416864 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:17.416835    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6_odh-ai-gateway-infra(4a1d34d8-4d62-4a52-bb97-ae9ea24eac00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00"
Oct 30 03:15:17.418262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.418239    4278 generic.go:356] "Generic (PLEG): container finished" podID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerID="20b2cd9b86bee06260d1f0b88fdd7d378c57315e5864f063605e991c99467c4b" exitCode=60
Oct 30 03:15:17.418355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.418271    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" event={"ID":"969c6501-88c9-4285-9b60-2d38d788bbd6","Type":"ContainerDied","Data":"20b2cd9b86bee06260d1f0b88fdd7d378c57315e5864f063605e991c99467c4b"}
Oct 30 03:15:17.418763 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.418742    4278 scope.go:122] "RemoveContainer" containerID="20b2cd9b86bee06260d1f0b88fdd7d378c57315e5864f063605e991c99467c4b"
Oct 30 03:15:17.419233 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:17.419207    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4_odh-ai-gateway-infra(969c6501-88c9-4285-9b60-2d38d788bbd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6"
Oct 30 03:15:17.420518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.420494    4278 generic.go:356] "Generic (PLEG): container finished" podID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerID="90714a00bf9feba3fd30110b1a58faaa3ca71d228e3f6739af7e10d1480d3479" exitCode=60
Oct 30 03:15:17.420614 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.420564    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" event={"ID":"c5968a2a-86ba-4717-8f0f-df111b79f123","Type":"ContainerDied","Data":"90714a00bf9feba3fd30110b1a58faaa3ca71d228e3f6739af7e10d1480d3479"}
Oct 30 03:15:17.420985 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.420970    4278 scope.go:122] "RemoveContainer" containerID="90714a00bf9feba3fd30110b1a58faaa3ca71d228e3f6739af7e10d1480d3479"
Oct 30 03:15:17.421333 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:17.421311    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf_odh-ai-gateway-infra(c5968a2a-86ba-4717-8f0f-df111b79f123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123"
Oct 30 03:15:17.432604 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.432585    4278 scope.go:122] "RemoveContainer" containerID="023932c4073a760e749b8c42efc246231562304dac1087ca36f281a3128d1975"
Oct 30 03:15:17.447616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.447593    4278 scope.go:122] "RemoveContainer" containerID="5eaf7d94586e5a5cbc224674a79529cf6875cc405975ac94a913fe0641be0e85"
Oct 30 03:15:17.463925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:17.463886    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"]
Oct 30 03:15:18.191964 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.191870    4278 scope.go:122] "RemoveContainer" containerID="81e9f74f8253e36dc415be57ccda6dd9f848176b5a24c2d4d57bdbd44066713f"
Oct 30 03:15:18.192127 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.192008    4278 scope.go:122] "RemoveContainer" containerID="953d6fa9016778fe8e26d02dcb90418a2311237a71d189ce1da8aa54a22794b1"
Oct 30 03:15:18.432296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.432262    4278 generic.go:356] "Generic (PLEG): container finished" podID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerID="acb3d104a531d8ab73944119fd383cf601ba905dd1fc7f66db4b74bf2ab302a4" exitCode=60
Oct 30 03:15:18.432534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.432336    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7" event={"ID":"a02df413-cc1b-4d7e-8209-16022147c7a1","Type":"ContainerDied","Data":"acb3d104a531d8ab73944119fd383cf601ba905dd1fc7f66db4b74bf2ab302a4"}
Oct 30 03:15:18.432534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.432409    4278 scope.go:122] "RemoveContainer" containerID="c391b6e1031c1c70f65e0016b6eccf77b0f5e13e5428356584ace6503021a818"
Oct 30 03:15:18.433032 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.433008    4278 scope.go:122] "RemoveContainer" containerID="acb3d104a531d8ab73944119fd383cf601ba905dd1fc7f66db4b74bf2ab302a4"
Oct 30 03:15:18.433567 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:18.433536    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7_odh-ai-gateway-infra(a02df413-cc1b-4d7e-8209-16022147c7a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1"
Oct 30 03:15:18.435013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.434988    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" event={"ID":"6c90ec76-2fe7-40e2-b88b-8b766b28e158","Type":"ContainerStarted","Data":"0664cf9e342baeb31fec7940a5718fafa545ec181641549c97a8cdd74c1592d5"}
Oct 30 03:15:18.440845 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.440820    4278 generic.go:356] "Generic (PLEG): container finished" podID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerID="e851e2dbdbc7ba215b154598d6c8a9474b4919560aad207a692c8a0938726380" exitCode=60
Oct 30 03:15:18.440947 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.440887    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx" event={"ID":"bfefef3e-400b-48d3-b4f8-bf432ee80c29","Type":"ContainerDied","Data":"e851e2dbdbc7ba215b154598d6c8a9474b4919560aad207a692c8a0938726380"}
Oct 30 03:15:18.441346 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.441327    4278 scope.go:122] "RemoveContainer" containerID="e851e2dbdbc7ba215b154598d6c8a9474b4919560aad207a692c8a0938726380"
Oct 30 03:15:18.441788 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:18.441765    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx_odh-ai-gateway-infra(bfefef3e-400b-48d3-b4f8-bf432ee80c29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29"
Oct 30 03:15:18.474743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.474399    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"]
Oct 30 03:15:18.566523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.565525    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"]
Oct 30 03:15:18.655278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.655253    4278 scope.go:122] "RemoveContainer" containerID="9535cd2bb9a993e773b973b31bd4ce8ee741f1563eafa83676c2a9b978b60406"
Oct 30 03:15:18.765853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.765236    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"
Oct 30 03:15:18.782662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.782630    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"
Oct 30 03:15:18.853014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.852961    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5968a2a-86ba-4717-8f0f-df111b79f123-kube-api-access-dnsbw\" (UniqueName: \"kubernetes.io/projected/c5968a2a-86ba-4717-8f0f-df111b79f123-kube-api-access-dnsbw\") pod \"c5968a2a-86ba-4717-8f0f-df111b79f123\" (UID: \"c5968a2a-86ba-4717-8f0f-df111b79f123\") "
Oct 30 03:15:18.853262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.853242    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-kube-api-access-xgjpw\" (UniqueName: \"kubernetes.io/projected/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-kube-api-access-xgjpw\") pod \"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00\" (UID: \"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00\") "
Oct 30 03:15:18.853416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.853398    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5968a2a-86ba-4717-8f0f-df111b79f123-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5968a2a-86ba-4717-8f0f-df111b79f123-maas-api-service-ca\") pod \"c5968a2a-86ba-4717-8f0f-df111b79f123\" (UID: \"c5968a2a-86ba-4717-8f0f-df111b79f123\") "
Oct 30 03:15:18.853554 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.853526    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-maas-api-service-ca\") pod \"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00\" (UID: \"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00\") "
Oct 30 03:15:18.853840 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.853807    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5968a2a-86ba-4717-8f0f-df111b79f123-maas-api-service-ca" pod "c5968a2a-86ba-4717-8f0f-df111b79f123" (UID: "c5968a2a-86ba-4717-8f0f-df111b79f123"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:18.855035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.854712    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-maas-api-service-ca" pod "4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" (UID: "4a1d34d8-4d62-4a52-bb97-ae9ea24eac00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:18.855035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.854786    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5968a2a-86ba-4717-8f0f-df111b79f123-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:18.862616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.861920    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5968a2a-86ba-4717-8f0f-df111b79f123-kube-api-access-dnsbw" pod "c5968a2a-86ba-4717-8f0f-df111b79f123" (UID: "c5968a2a-86ba-4717-8f0f-df111b79f123"). InnerVolumeSpecName "kube-api-access-dnsbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:18.864980 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.864880    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-kube-api-access-xgjpw" pod "4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" (UID: "4a1d34d8-4d62-4a52-bb97-ae9ea24eac00"). InnerVolumeSpecName "kube-api-access-xgjpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:18.886056 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.885657    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"
Oct 30 03:15:18.955921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.955816    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969c6501-88c9-4285-9b60-2d38d788bbd6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c6501-88c9-4285-9b60-2d38d788bbd6-maas-api-service-ca\") pod \"969c6501-88c9-4285-9b60-2d38d788bbd6\" (UID: \"969c6501-88c9-4285-9b60-2d38d788bbd6\") "
Oct 30 03:15:18.955921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.955873    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969c6501-88c9-4285-9b60-2d38d788bbd6-kube-api-access-f6xbn\" (UniqueName: \"kubernetes.io/projected/969c6501-88c9-4285-9b60-2d38d788bbd6-kube-api-access-f6xbn\") pod \"969c6501-88c9-4285-9b60-2d38d788bbd6\" (UID: \"969c6501-88c9-4285-9b60-2d38d788bbd6\") "
Oct 30 03:15:18.956296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.956267    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969c6501-88c9-4285-9b60-2d38d788bbd6-maas-api-service-ca" pod "969c6501-88c9-4285-9b60-2d38d788bbd6" (UID: "969c6501-88c9-4285-9b60-2d38d788bbd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:18.956774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.956750    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgjpw\" (UniqueName: \"kubernetes.io/projected/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-kube-api-access-xgjpw\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:18.956774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.956774    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:18.956953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.956786    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c6501-88c9-4285-9b60-2d38d788bbd6-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:18.956953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.956795    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnsbw\" (UniqueName: \"kubernetes.io/projected/c5968a2a-86ba-4717-8f0f-df111b79f123-kube-api-access-dnsbw\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:18.958189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:18.958162    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969c6501-88c9-4285-9b60-2d38d788bbd6-kube-api-access-f6xbn" pod "969c6501-88c9-4285-9b60-2d38d788bbd6" (UID: "969c6501-88c9-4285-9b60-2d38d788bbd6"). InnerVolumeSpecName "kube-api-access-f6xbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:19.059331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.059291    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6xbn\" (UniqueName: \"kubernetes.io/projected/969c6501-88c9-4285-9b60-2d38d788bbd6-kube-api-access-f6xbn\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:19.192545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.192508    4278 scope.go:122] "RemoveContainer" containerID="ccf09e796dd3f7dd9d06ccfd92bbfdefca1ed2d16a7e37b47cfefd9524437f5f"
Oct 30 03:15:19.192973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.192852    4278 scope.go:122] "RemoveContainer" containerID="94d9d9e4f21376005dfccc935467fd73d440bc82f8c11d30814f779f6a88a0fd"
Oct 30 03:15:19.193268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.193238    4278 scope.go:122] "RemoveContainer" containerID="fb0f4445b5266656f37cc38ea603f6c310d09cee7606753905425027417ae559"
Oct 30 03:15:19.193468 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.193338    4278 scope.go:122] "RemoveContainer" containerID="cade8d0f2a698961c9697ffed7a94c8f89b69c01a67c63adff55b66ea358927d"
Oct 30 03:15:19.193717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.193467    4278 scope.go:122] "RemoveContainer" containerID="aa669a9b04391434c65fec795ff96e1a3bd5252acb7d7677b7f1d17d9704068b"
Oct 30 03:15:19.454106 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.454072    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"
Oct 30 03:15:19.454271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.454096    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6" event={"ID":"4a1d34d8-4d62-4a52-bb97-ae9ea24eac00","Type":"ContainerDied","Data":"9903b4689f2e50514393d00c46c775069e384f58976b00fc7e149b117e5c9511"}
Oct 30 03:15:19.454271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.454175    4278 scope.go:122] "RemoveContainer" containerID="e0b43592cab04e54b17ca90ae4405087764cc4f727873b633a5b1fb99bf8b169"
Oct 30 03:15:19.455998 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.455970    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4" event={"ID":"969c6501-88c9-4285-9b60-2d38d788bbd6","Type":"ContainerDied","Data":"f6a91ce9fabbb54384984de3e9380b34576bb28567736ccf5194ae0569edf742"}
Oct 30 03:15:19.456114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.455982    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"
Oct 30 03:15:19.463884 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.463859    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"
Oct 30 03:15:19.464118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.463881    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf" event={"ID":"c5968a2a-86ba-4717-8f0f-df111b79f123","Type":"ContainerDied","Data":"4be2cdceafac7b6d86ad7f3ef738d9f27d2504d18b397701ae2050ef299f2746"}
Oct 30 03:15:19.468093 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.468050    4278 generic.go:356] "Generic (PLEG): container finished" podID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerID="312239a6cad324b2ce0eb5116355955c481bf44e026a588d15bb78e32ed82a5b" exitCode=60
Oct 30 03:15:19.468221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.468131    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" event={"ID":"5103b31a-7eaf-4364-9af6-1b7eb34e1d40","Type":"ContainerDied","Data":"312239a6cad324b2ce0eb5116355955c481bf44e026a588d15bb78e32ed82a5b"}
Oct 30 03:15:19.468688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.468665    4278 scope.go:122] "RemoveContainer" containerID="312239a6cad324b2ce0eb5116355955c481bf44e026a588d15bb78e32ed82a5b"
Oct 30 03:15:19.469065 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:19.469039    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j_odh-ai-gateway-infra(5103b31a-7eaf-4364-9af6-1b7eb34e1d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40"
Oct 30 03:15:19.474099 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.474032    4278 generic.go:356] "Generic (PLEG): container finished" podID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerID="0664cf9e342baeb31fec7940a5718fafa545ec181641549c97a8cdd74c1592d5" exitCode=60
Oct 30 03:15:19.474193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.474099    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" event={"ID":"6c90ec76-2fe7-40e2-b88b-8b766b28e158","Type":"ContainerDied","Data":"0664cf9e342baeb31fec7940a5718fafa545ec181641549c97a8cdd74c1592d5"}
Oct 30 03:15:19.474560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.474538    4278 scope.go:122] "RemoveContainer" containerID="0664cf9e342baeb31fec7940a5718fafa545ec181641549c97a8cdd74c1592d5"
Oct 30 03:15:19.474970 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:19.474945    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc_odh-ai-gateway-infra(6c90ec76-2fe7-40e2-b88b-8b766b28e158)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158"
Oct 30 03:15:19.476907 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.476882    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws" event={"ID":"9e75f3b5-4d25-4fed-91f1-99a41800f320","Type":"ContainerStarted","Data":"6084c1cef8c76c3ae45fb7ec7717353c0ed78164ed93cc6c72e72ccbadaf472a"}
Oct 30 03:15:19.483797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.482597    4278 scope.go:122] "RemoveContainer" containerID="20b2cd9b86bee06260d1f0b88fdd7d378c57315e5864f063605e991c99467c4b"
Oct 30 03:15:19.487075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.487035    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"]
Oct 30 03:15:19.496601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.496579    4278 scope.go:122] "RemoveContainer" containerID="90714a00bf9feba3fd30110b1a58faaa3ca71d228e3f6739af7e10d1480d3479"
Oct 30 03:15:19.510685 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.510659    4278 scope.go:122] "RemoveContainer" containerID="953d6fa9016778fe8e26d02dcb90418a2311237a71d189ce1da8aa54a22794b1"
Oct 30 03:15:19.534362 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.534337    4278 scope.go:122] "RemoveContainer" containerID="9464e3e3200b0d01f5dc529645fdac2137a4a2a8f5ba796336f6fe97f077a9a1"
Oct 30 03:15:19.535415 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.535391    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"]
Oct 30 03:15:19.626151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.626115    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"]
Oct 30 03:15:19.638077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.638040    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"]
Oct 30 03:15:19.642663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.642634    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e2727-29888835-2f7c6"]
Oct 30 03:15:19.684593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.684558    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"]
Oct 30 03:15:19.694538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.694505    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16df10-29888835-xxrxf"]
Oct 30 03:15:19.795025 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.794819    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"]
Oct 30 03:15:19.809627 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.806715    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-810119-29888835-4l2z4"]
Oct 30 03:15:19.900414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.900353    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws" podStartSLOduration=19.900334324 podStartE2EDuration="19.900334324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:19.851060565 +0000 UTC m=+2057.775268757" watchObservedRunningTime="2026-10-30 03:15:19.900334324 +0000 UTC m=+2057.824542513"
Oct 30 03:15:19.969893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.969862    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"
Oct 30 03:15:19.994873 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:19.994847    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"
Oct 30 03:15:20.080631 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.080295    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02df413-cc1b-4d7e-8209-16022147c7a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02df413-cc1b-4d7e-8209-16022147c7a1-maas-api-service-ca\") pod \"a02df413-cc1b-4d7e-8209-16022147c7a1\" (UID: \"a02df413-cc1b-4d7e-8209-16022147c7a1\") "
Oct 30 03:15:20.080631 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.080495    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfefef3e-400b-48d3-b4f8-bf432ee80c29-kube-api-access-ffvth\" (UniqueName: \"kubernetes.io/projected/bfefef3e-400b-48d3-b4f8-bf432ee80c29-kube-api-access-ffvth\") pod \"bfefef3e-400b-48d3-b4f8-bf432ee80c29\" (UID: \"bfefef3e-400b-48d3-b4f8-bf432ee80c29\") "
Oct 30 03:15:20.080631 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.080620    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02df413-cc1b-4d7e-8209-16022147c7a1-kube-api-access-j8gzz\" (UniqueName: \"kubernetes.io/projected/a02df413-cc1b-4d7e-8209-16022147c7a1-kube-api-access-j8gzz\") pod \"a02df413-cc1b-4d7e-8209-16022147c7a1\" (UID: \"a02df413-cc1b-4d7e-8209-16022147c7a1\") "
Oct 30 03:15:20.080985 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.080816    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02df413-cc1b-4d7e-8209-16022147c7a1-maas-api-service-ca" pod "a02df413-cc1b-4d7e-8209-16022147c7a1" (UID: "a02df413-cc1b-4d7e-8209-16022147c7a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.080985 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.080841    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfefef3e-400b-48d3-b4f8-bf432ee80c29-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfefef3e-400b-48d3-b4f8-bf432ee80c29-maas-api-service-ca\") pod \"bfefef3e-400b-48d3-b4f8-bf432ee80c29\" (UID: \"bfefef3e-400b-48d3-b4f8-bf432ee80c29\") "
Oct 30 03:15:20.082312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.082279    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02df413-cc1b-4d7e-8209-16022147c7a1-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.082707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.082681    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfefef3e-400b-48d3-b4f8-bf432ee80c29-maas-api-service-ca" pod "bfefef3e-400b-48d3-b4f8-bf432ee80c29" (UID: "bfefef3e-400b-48d3-b4f8-bf432ee80c29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.083638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.083610    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02df413-cc1b-4d7e-8209-16022147c7a1-kube-api-access-j8gzz" pod "a02df413-cc1b-4d7e-8209-16022147c7a1" (UID: "a02df413-cc1b-4d7e-8209-16022147c7a1"). InnerVolumeSpecName "kube-api-access-j8gzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.083723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.083661    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfefef3e-400b-48d3-b4f8-bf432ee80c29-kube-api-access-ffvth" pod "bfefef3e-400b-48d3-b4f8-bf432ee80c29" (UID: "bfefef3e-400b-48d3-b4f8-bf432ee80c29"). InnerVolumeSpecName "kube-api-access-ffvth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.184172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.184123    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffvth\" (UniqueName: \"kubernetes.io/projected/bfefef3e-400b-48d3-b4f8-bf432ee80c29-kube-api-access-ffvth\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.184172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.184159    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8gzz\" (UniqueName: \"kubernetes.io/projected/a02df413-cc1b-4d7e-8209-16022147c7a1-kube-api-access-j8gzz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.184172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.184174    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfefef3e-400b-48d3-b4f8-bf432ee80c29-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.201711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.201676    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" path="/var/lib/kubelet/pods/4a1d34d8-4d62-4a52-bb97-ae9ea24eac00/volumes"
Oct 30 03:15:20.202244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.202227    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6" path="/var/lib/kubelet/pods/969c6501-88c9-4285-9b60-2d38d788bbd6/volumes"
Oct 30 03:15:20.202687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.202670    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123" path="/var/lib/kubelet/pods/c5968a2a-86ba-4717-8f0f-df111b79f123/volumes"
Oct 30 03:15:20.489523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.489485    4278 generic.go:356] "Generic (PLEG): container finished" podID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerID="0ef96bdf513319c7498e5c26f0a5a850e79faade9662b3bd19a07d084be01bc5" exitCode=60
Oct 30 03:15:20.489694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.489554    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" event={"ID":"fed730a6-41b7-4cfc-af55-ae03dbc48d38","Type":"ContainerDied","Data":"0ef96bdf513319c7498e5c26f0a5a850e79faade9662b3bd19a07d084be01bc5"}
Oct 30 03:15:20.489694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.489623    4278 scope.go:122] "RemoveContainer" containerID="cade8d0f2a698961c9697ffed7a94c8f89b69c01a67c63adff55b66ea358927d"
Oct 30 03:15:20.490185 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.490163    4278 scope.go:122] "RemoveContainer" containerID="0ef96bdf513319c7498e5c26f0a5a850e79faade9662b3bd19a07d084be01bc5"
Oct 30 03:15:20.490744 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:20.490714    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws_odh-ai-gateway-infra(fed730a6-41b7-4cfc-af55-ae03dbc48d38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38"
Oct 30 03:15:20.495371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.495343    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx" event={"ID":"bfefef3e-400b-48d3-b4f8-bf432ee80c29","Type":"ContainerDied","Data":"16d7613c04dfcce476aea18f17a44ba11e711e884c3ef6d776bcaf5d3b821ba7"}
Oct 30 03:15:20.495535 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.495359    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"
Oct 30 03:15:20.499524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.499490    4278 generic.go:356] "Generic (PLEG): container finished" podID="ea6c297a-e973-4e59-859f-5c4862900275" containerID="1d6cde6aca877a1401435efdfd681c29cb8b8f127105331fc4914e72e58eb96d" exitCode=60
Oct 30 03:15:20.499637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.499608    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" event={"ID":"ea6c297a-e973-4e59-859f-5c4862900275","Type":"ContainerDied","Data":"1d6cde6aca877a1401435efdfd681c29cb8b8f127105331fc4914e72e58eb96d"}
Oct 30 03:15:20.502517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.502497    4278 scope.go:122] "RemoveContainer" containerID="1d6cde6aca877a1401435efdfd681c29cb8b8f127105331fc4914e72e58eb96d"
Oct 30 03:15:20.502981 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:20.502956    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x_odh-ai-gateway-infra(ea6c297a-e973-4e59-859f-5c4862900275)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" podUID="ea6c297a-e973-4e59-859f-5c4862900275"
Oct 30 03:15:20.505404 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.505382    4278 generic.go:356] "Generic (PLEG): container finished" podID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerID="61b3c3d85a6a750a42523388cf153136a69f40f9f5b64e9ac04645a452599557" exitCode=60
Oct 30 03:15:20.505504 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.505487    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" event={"ID":"8223ce14-326f-40bd-bb96-dc1b24bb4fd0","Type":"ContainerDied","Data":"61b3c3d85a6a750a42523388cf153136a69f40f9f5b64e9ac04645a452599557"}
Oct 30 03:15:20.505932 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.505915    4278 scope.go:122] "RemoveContainer" containerID="61b3c3d85a6a750a42523388cf153136a69f40f9f5b64e9ac04645a452599557"
Oct 30 03:15:20.506258 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:20.506239    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj_odh-ai-gateway-infra(8223ce14-326f-40bd-bb96-dc1b24bb4fd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0"
Oct 30 03:15:20.520807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.520771    4278 generic.go:356] "Generic (PLEG): container finished" podID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerID="483411bd1c3f400fe52b2a1c10f3bf557361df081e29e23b3c207879992a74b6" exitCode=60
Oct 30 03:15:20.520968 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.520889    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" event={"ID":"0d2f4868-04fa-430b-93bc-9644c7a526cd","Type":"ContainerDied","Data":"483411bd1c3f400fe52b2a1c10f3bf557361df081e29e23b3c207879992a74b6"}
Oct 30 03:15:20.521505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.521481    4278 scope.go:122] "RemoveContainer" containerID="483411bd1c3f400fe52b2a1c10f3bf557361df081e29e23b3c207879992a74b6"
Oct 30 03:15:20.522000 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:20.521972    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb_odh-ai-gateway-infra(0d2f4868-04fa-430b-93bc-9644c7a526cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd"
Oct 30 03:15:20.523926 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.523895    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7" event={"ID":"a02df413-cc1b-4d7e-8209-16022147c7a1","Type":"ContainerDied","Data":"18746edfbb9efc4aea4be6a80ef9285186d1711abebb58c60c902c461e0e042f"}
Oct 30 03:15:20.524057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.524008    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"
Oct 30 03:15:20.530082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.530056    4278 generic.go:356] "Generic (PLEG): container finished" podID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerID="6084c1cef8c76c3ae45fb7ec7717353c0ed78164ed93cc6c72e72ccbadaf472a" exitCode=60
Oct 30 03:15:20.530288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.530139    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws" event={"ID":"9e75f3b5-4d25-4fed-91f1-99a41800f320","Type":"ContainerDied","Data":"6084c1cef8c76c3ae45fb7ec7717353c0ed78164ed93cc6c72e72ccbadaf472a"}
Oct 30 03:15:20.530758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.530729    4278 scope.go:122] "RemoveContainer" containerID="6084c1cef8c76c3ae45fb7ec7717353c0ed78164ed93cc6c72e72ccbadaf472a"
Oct 30 03:15:20.531226 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:20.531189    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws_odh-ai-gateway-infra(9e75f3b5-4d25-4fed-91f1-99a41800f320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320"
Oct 30 03:15:20.533017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.532990    4278 generic.go:356] "Generic (PLEG): container finished" podID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerID="62f3490efc3fe72f674a7f7db2fa808f1c569f98e06352e9b073a8285bbf6d72" exitCode=60
Oct 30 03:15:20.533116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.533064    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" event={"ID":"8e6f2b85-9672-4fc7-8a8a-0393e7886187","Type":"ContainerDied","Data":"62f3490efc3fe72f674a7f7db2fa808f1c569f98e06352e9b073a8285bbf6d72"}
Oct 30 03:15:20.533576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.533556    4278 scope.go:122] "RemoveContainer" containerID="62f3490efc3fe72f674a7f7db2fa808f1c569f98e06352e9b073a8285bbf6d72"
Oct 30 03:15:20.533942 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:20.533920    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb_odh-ai-gateway-infra(8e6f2b85-9672-4fc7-8a8a-0393e7886187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187"
Oct 30 03:15:20.546763 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.546724    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"]
Oct 30 03:15:20.576436 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.574734    4278 scope.go:122] "RemoveContainer" containerID="e851e2dbdbc7ba215b154598d6c8a9474b4919560aad207a692c8a0938726380"
Oct 30 03:15:20.622340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.622279    4278 scope.go:122] "RemoveContainer" containerID="aa669a9b04391434c65fec795ff96e1a3bd5252acb7d7677b7f1d17d9704068b"
Oct 30 03:15:20.650690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.650661    4278 scope.go:122] "RemoveContainer" containerID="ccf09e796dd3f7dd9d06ccfd92bbfdefca1ed2d16a7e37b47cfefd9524437f5f"
Oct 30 03:15:20.672931 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.672901    4278 scope.go:122] "RemoveContainer" containerID="94d9d9e4f21376005dfccc935467fd73d440bc82f8c11d30814f779f6a88a0fd"
Oct 30 03:15:20.696602 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.696573    4278 scope.go:122] "RemoveContainer" containerID="acb3d104a531d8ab73944119fd383cf601ba905dd1fc7f66db4b74bf2ab302a4"
Oct 30 03:15:20.720733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.720706    4278 scope.go:122] "RemoveContainer" containerID="81e9f74f8253e36dc415be57ccda6dd9f848176b5a24c2d4d57bdbd44066713f"
Oct 30 03:15:20.738331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.738295    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"]
Oct 30 03:15:20.750625 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.750597    4278 scope.go:122] "RemoveContainer" containerID="fb0f4445b5266656f37cc38ea603f6c310d09cee7606753905425027417ae559"
Oct 30 03:15:20.758865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.758820    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888835-4dxl7"]
Oct 30 03:15:20.838319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.838293    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"
Oct 30 03:15:20.842303 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.842278    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"
Oct 30 03:15:20.856237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.856203    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"]
Oct 30 03:15:20.999796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.999705    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c90ec76-2fe7-40e2-b88b-8b766b28e158-kube-api-access-8xznh\" (UniqueName: \"kubernetes.io/projected/6c90ec76-2fe7-40e2-b88b-8b766b28e158-kube-api-access-8xznh\") pod \"6c90ec76-2fe7-40e2-b88b-8b766b28e158\" (UID: \"6c90ec76-2fe7-40e2-b88b-8b766b28e158\") "
Oct 30 03:15:20.999975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.999844    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-maas-api-service-ca\") pod \"5103b31a-7eaf-4364-9af6-1b7eb34e1d40\" (UID: \"5103b31a-7eaf-4364-9af6-1b7eb34e1d40\") "
Oct 30 03:15:20.999975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:20.999902    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c90ec76-2fe7-40e2-b88b-8b766b28e158-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c90ec76-2fe7-40e2-b88b-8b766b28e158-maas-api-service-ca\") pod \"6c90ec76-2fe7-40e2-b88b-8b766b28e158\" (UID: \"6c90ec76-2fe7-40e2-b88b-8b766b28e158\") "
Oct 30 03:15:21.000091 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.000055    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-kube-api-access-2ftxp\" (UniqueName: \"kubernetes.io/projected/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-kube-api-access-2ftxp\") pod \"5103b31a-7eaf-4364-9af6-1b7eb34e1d40\" (UID: \"5103b31a-7eaf-4364-9af6-1b7eb34e1d40\") "
Oct 30 03:15:21.000270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.000245    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-maas-api-service-ca" pod "5103b31a-7eaf-4364-9af6-1b7eb34e1d40" (UID: "5103b31a-7eaf-4364-9af6-1b7eb34e1d40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:21.000339 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.000318    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c90ec76-2fe7-40e2-b88b-8b766b28e158-maas-api-service-ca" pod "6c90ec76-2fe7-40e2-b88b-8b766b28e158" (UID: "6c90ec76-2fe7-40e2-b88b-8b766b28e158"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:21.000908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.000881    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:21.000908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.000907    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c90ec76-2fe7-40e2-b88b-8b766b28e158-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:21.002713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.002277    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c90ec76-2fe7-40e2-b88b-8b766b28e158-kube-api-access-8xznh" pod "6c90ec76-2fe7-40e2-b88b-8b766b28e158" (UID: "6c90ec76-2fe7-40e2-b88b-8b766b28e158"). InnerVolumeSpecName "kube-api-access-8xznh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:21.002713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.002687    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-kube-api-access-2ftxp" pod "5103b31a-7eaf-4364-9af6-1b7eb34e1d40" (UID: "5103b31a-7eaf-4364-9af6-1b7eb34e1d40"). InnerVolumeSpecName "kube-api-access-2ftxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:21.004656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.004632    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"]
Oct 30 03:15:21.009681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.009654    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-360818-29888835-fnvrx"]
Oct 30 03:15:21.102225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.102179    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ftxp\" (UniqueName: \"kubernetes.io/projected/5103b31a-7eaf-4364-9af6-1b7eb34e1d40-kube-api-access-2ftxp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:21.102225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.102225    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xznh\" (UniqueName: \"kubernetes.io/projected/6c90ec76-2fe7-40e2-b88b-8b766b28e158-kube-api-access-8xznh\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:21.576843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.576797    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j" event={"ID":"5103b31a-7eaf-4364-9af6-1b7eb34e1d40","Type":"ContainerDied","Data":"b01dd662818b22078535dd5d152dbf11b722d310c76768b428133b733ecd036f"}
Oct 30 03:15:21.577495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.577408    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"
Oct 30 03:15:21.577689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.577672    4278 scope.go:122] "RemoveContainer" containerID="312239a6cad324b2ce0eb5116355955c481bf44e026a588d15bb78e32ed82a5b"
Oct 30 03:15:21.587046 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.587010    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc" event={"ID":"6c90ec76-2fe7-40e2-b88b-8b766b28e158","Type":"ContainerDied","Data":"f1d1f2fad2b3dd124185a27ee36ce57d7cfb4e2de99e1f18c31da475817a2d00"}
Oct 30 03:15:21.587167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.587144    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"
Oct 30 03:15:21.595215 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.595167    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"]
Oct 30 03:15:21.601446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.601404    4278 scope.go:122] "RemoveContainer" containerID="0664cf9e342baeb31fec7940a5718fafa545ec181641549c97a8cdd74c1592d5"
Oct 30 03:15:21.650726 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.650690    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"]
Oct 30 03:15:21.655918 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.655888    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1fce3-29888835-wf62j"]
Oct 30 03:15:21.713363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.713328    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"]
Oct 30 03:15:21.810857 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.810829    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"]
Oct 30 03:15:21.829602 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.828623    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"]
Oct 30 03:15:21.834193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.834166    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"
Oct 30 03:15:21.836869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.836842    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9d3552-29888835-sdxlc"]
Oct 30 03:15:21.854665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.854634    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"]
Oct 30 03:15:21.920671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.920332    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"
Oct 30 03:15:21.934455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.933420    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e75f3b5-4d25-4fed-91f1-99a41800f320-kube-api-access-4p7lf\" (UniqueName: \"kubernetes.io/projected/9e75f3b5-4d25-4fed-91f1-99a41800f320-kube-api-access-4p7lf\") pod \"9e75f3b5-4d25-4fed-91f1-99a41800f320\" (UID: \"9e75f3b5-4d25-4fed-91f1-99a41800f320\") "
Oct 30 03:15:21.934455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.933614    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e75f3b5-4d25-4fed-91f1-99a41800f320-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e75f3b5-4d25-4fed-91f1-99a41800f320-maas-api-service-ca\") pod \"9e75f3b5-4d25-4fed-91f1-99a41800f320\" (UID: \"9e75f3b5-4d25-4fed-91f1-99a41800f320\") "
Oct 30 03:15:21.934455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.933719    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed730a6-41b7-4cfc-af55-ae03dbc48d38-kube-api-access-5kq4z\" (UniqueName: \"kubernetes.io/projected/fed730a6-41b7-4cfc-af55-ae03dbc48d38-kube-api-access-5kq4z\") pod \"fed730a6-41b7-4cfc-af55-ae03dbc48d38\" (UID: \"fed730a6-41b7-4cfc-af55-ae03dbc48d38\") "
Oct 30 03:15:21.934455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.933861    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed730a6-41b7-4cfc-af55-ae03dbc48d38-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed730a6-41b7-4cfc-af55-ae03dbc48d38-maas-api-service-ca\") pod \"fed730a6-41b7-4cfc-af55-ae03dbc48d38\" (UID: \"fed730a6-41b7-4cfc-af55-ae03dbc48d38\") "
Oct 30 03:15:21.934455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.934061    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e75f3b5-4d25-4fed-91f1-99a41800f320-maas-api-service-ca" pod "9e75f3b5-4d25-4fed-91f1-99a41800f320" (UID: "9e75f3b5-4d25-4fed-91f1-99a41800f320"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:21.934455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.934274    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed730a6-41b7-4cfc-af55-ae03dbc48d38-maas-api-service-ca" pod "fed730a6-41b7-4cfc-af55-ae03dbc48d38" (UID: "fed730a6-41b7-4cfc-af55-ae03dbc48d38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:21.934897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.934655    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e75f3b5-4d25-4fed-91f1-99a41800f320-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:21.934897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.934675    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed730a6-41b7-4cfc-af55-ae03dbc48d38-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:21.936598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.936550    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e75f3b5-4d25-4fed-91f1-99a41800f320-kube-api-access-4p7lf" pod "9e75f3b5-4d25-4fed-91f1-99a41800f320" (UID: "9e75f3b5-4d25-4fed-91f1-99a41800f320"). InnerVolumeSpecName "kube-api-access-4p7lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:21.936904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.936884    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed730a6-41b7-4cfc-af55-ae03dbc48d38-kube-api-access-5kq4z" pod "fed730a6-41b7-4cfc-af55-ae03dbc48d38" (UID: "fed730a6-41b7-4cfc-af55-ae03dbc48d38"). InnerVolumeSpecName "kube-api-access-5kq4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:21.989894 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:21.989855    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"]
Oct 30 03:15:22.036788 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.036757    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p7lf\" (UniqueName: \"kubernetes.io/projected/9e75f3b5-4d25-4fed-91f1-99a41800f320-kube-api-access-4p7lf\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.037007 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.036987    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kq4z\" (UniqueName: \"kubernetes.io/projected/fed730a6-41b7-4cfc-af55-ae03dbc48d38-kube-api-access-5kq4z\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.045452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.045335    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"
Oct 30 03:15:22.118049 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.118015    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"
Oct 30 03:15:22.138601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.138253    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2f4868-04fa-430b-93bc-9644c7a526cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2f4868-04fa-430b-93bc-9644c7a526cd-maas-api-service-ca\") pod \"0d2f4868-04fa-430b-93bc-9644c7a526cd\" (UID: \"0d2f4868-04fa-430b-93bc-9644c7a526cd\") "
Oct 30 03:15:22.138601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.138474    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6c297a-e973-4e59-859f-5c4862900275-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6c297a-e973-4e59-859f-5c4862900275-maas-api-service-ca\") pod \"ea6c297a-e973-4e59-859f-5c4862900275\" (UID: \"ea6c297a-e973-4e59-859f-5c4862900275\") "
Oct 30 03:15:22.138601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.138513    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2f4868-04fa-430b-93bc-9644c7a526cd-kube-api-access-9hc7s\" (UniqueName: \"kubernetes.io/projected/0d2f4868-04fa-430b-93bc-9644c7a526cd-kube-api-access-9hc7s\") pod \"0d2f4868-04fa-430b-93bc-9644c7a526cd\" (UID: \"0d2f4868-04fa-430b-93bc-9644c7a526cd\") "
Oct 30 03:15:22.138816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.138707    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6c297a-e973-4e59-859f-5c4862900275-kube-api-access-6tmb6\" (UniqueName: \"kubernetes.io/projected/ea6c297a-e973-4e59-859f-5c4862900275-kube-api-access-6tmb6\") pod \"ea6c297a-e973-4e59-859f-5c4862900275\" (UID: \"ea6c297a-e973-4e59-859f-5c4862900275\") "
Oct 30 03:15:22.142018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.141456    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2f4868-04fa-430b-93bc-9644c7a526cd-maas-api-service-ca" pod "0d2f4868-04fa-430b-93bc-9644c7a526cd" (UID: "0d2f4868-04fa-430b-93bc-9644c7a526cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:22.142018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.141877    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6c297a-e973-4e59-859f-5c4862900275-maas-api-service-ca" pod "ea6c297a-e973-4e59-859f-5c4862900275" (UID: "ea6c297a-e973-4e59-859f-5c4862900275"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:22.142578 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.142551    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6c297a-e973-4e59-859f-5c4862900275-kube-api-access-6tmb6" pod "ea6c297a-e973-4e59-859f-5c4862900275" (UID: "ea6c297a-e973-4e59-859f-5c4862900275"). InnerVolumeSpecName "kube-api-access-6tmb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:22.144498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.144470    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2f4868-04fa-430b-93bc-9644c7a526cd-kube-api-access-9hc7s" pod "0d2f4868-04fa-430b-93bc-9644c7a526cd" (UID: "0d2f4868-04fa-430b-93bc-9644c7a526cd"). InnerVolumeSpecName "kube-api-access-9hc7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:22.187776 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.187744    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"
Oct 30 03:15:22.207276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.207240    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" path="/var/lib/kubelet/pods/5103b31a-7eaf-4364-9af6-1b7eb34e1d40/volumes"
Oct 30 03:15:22.207866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.207848    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" path="/var/lib/kubelet/pods/6c90ec76-2fe7-40e2-b88b-8b766b28e158/volumes"
Oct 30 03:15:22.208304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.208290    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1" path="/var/lib/kubelet/pods/a02df413-cc1b-4d7e-8209-16022147c7a1/volumes"
Oct 30 03:15:22.209066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.209049    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" path="/var/lib/kubelet/pods/bfefef3e-400b-48d3-b4f8-bf432ee80c29/volumes"
Oct 30 03:15:22.241246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.240685    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6f2b85-9672-4fc7-8a8a-0393e7886187-kube-api-access-7gzjx\" (UniqueName: \"kubernetes.io/projected/8e6f2b85-9672-4fc7-8a8a-0393e7886187-kube-api-access-7gzjx\") pod \"8e6f2b85-9672-4fc7-8a8a-0393e7886187\" (UID: \"8e6f2b85-9672-4fc7-8a8a-0393e7886187\") "
Oct 30 03:15:22.241246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.240928    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6f2b85-9672-4fc7-8a8a-0393e7886187-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6f2b85-9672-4fc7-8a8a-0393e7886187-maas-api-service-ca\") pod \"8e6f2b85-9672-4fc7-8a8a-0393e7886187\" (UID: \"8e6f2b85-9672-4fc7-8a8a-0393e7886187\") "
Oct 30 03:15:22.242252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.242209    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2f4868-04fa-430b-93bc-9644c7a526cd-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.242252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.242237    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6c297a-e973-4e59-859f-5c4862900275-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.242638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.242268    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hc7s\" (UniqueName: \"kubernetes.io/projected/0d2f4868-04fa-430b-93bc-9644c7a526cd-kube-api-access-9hc7s\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.242638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.242283    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmb6\" (UniqueName: \"kubernetes.io/projected/ea6c297a-e973-4e59-859f-5c4862900275-kube-api-access-6tmb6\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.242735 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.242707    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6f2b85-9672-4fc7-8a8a-0393e7886187-maas-api-service-ca" pod "8e6f2b85-9672-4fc7-8a8a-0393e7886187" (UID: "8e6f2b85-9672-4fc7-8a8a-0393e7886187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:22.249683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.249656    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6f2b85-9672-4fc7-8a8a-0393e7886187-kube-api-access-7gzjx" pod "8e6f2b85-9672-4fc7-8a8a-0393e7886187" (UID: "8e6f2b85-9672-4fc7-8a8a-0393e7886187"). InnerVolumeSpecName "kube-api-access-7gzjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:22.275975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.275947    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"
Oct 30 03:15:22.343302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.343263    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-kube-api-access-vg58m\" (UniqueName: \"kubernetes.io/projected/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-kube-api-access-vg58m\") pod \"8223ce14-326f-40bd-bb96-dc1b24bb4fd0\" (UID: \"8223ce14-326f-40bd-bb96-dc1b24bb4fd0\") "
Oct 30 03:15:22.343540 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.343353    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-maas-api-service-ca\") pod \"8223ce14-326f-40bd-bb96-dc1b24bb4fd0\" (UID: \"8223ce14-326f-40bd-bb96-dc1b24bb4fd0\") "
Oct 30 03:15:22.343919 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.343885    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-maas-api-service-ca" pod "8223ce14-326f-40bd-bb96-dc1b24bb4fd0" (UID: "8223ce14-326f-40bd-bb96-dc1b24bb4fd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:22.344312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.344290    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.344505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.344317    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzjx\" (UniqueName: \"kubernetes.io/projected/8e6f2b85-9672-4fc7-8a8a-0393e7886187-kube-api-access-7gzjx\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.344505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.344334    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6f2b85-9672-4fc7-8a8a-0393e7886187-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.345751 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.345723    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-kube-api-access-vg58m" pod "8223ce14-326f-40bd-bb96-dc1b24bb4fd0" (UID: "8223ce14-326f-40bd-bb96-dc1b24bb4fd0"). InnerVolumeSpecName "kube-api-access-vg58m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:22.445780 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.445751    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg58m\" (UniqueName: \"kubernetes.io/projected/8223ce14-326f-40bd-bb96-dc1b24bb4fd0-kube-api-access-vg58m\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.607200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.607168    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"
Oct 30 03:15:22.607200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.607191    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws" event={"ID":"fed730a6-41b7-4cfc-af55-ae03dbc48d38","Type":"ContainerDied","Data":"f8eff2a05b5745a865ba552e82f2bdb86cbcbdd91ea40d7c657d0f438ea08e3d"}
Oct 30 03:15:22.607835 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.607262    4278 scope.go:122] "RemoveContainer" containerID="0ef96bdf513319c7498e5c26f0a5a850e79faade9662b3bd19a07d084be01bc5"
Oct 30 03:15:22.610572 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.610536    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x" event={"ID":"ea6c297a-e973-4e59-859f-5c4862900275","Type":"ContainerDied","Data":"f5723e36a9c24219e59280c4ba759a6653350df170e0c0b0a1e1b293227f06cb"}
Oct 30 03:15:22.617599 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.617563    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj" event={"ID":"8223ce14-326f-40bd-bb96-dc1b24bb4fd0","Type":"ContainerDied","Data":"48cff14b32d51b0c171802dfa504a916d402f40363105b4e79916c6a94548885"}
Oct 30 03:15:22.617762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.616533    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"
Oct 30 03:15:22.617951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.610567    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"
Oct 30 03:15:22.626613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.626589    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"
Oct 30 03:15:22.627955 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.626693    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb" event={"ID":"0d2f4868-04fa-430b-93bc-9644c7a526cd","Type":"ContainerDied","Data":"4eab06c511e03b8608acf1fbe7db2f5e881a2d0c98e8ed8d6403bf11d0051ed6"}
Oct 30 03:15:22.633906 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.633795    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"
Oct 30 03:15:22.633906 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.633804    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws" event={"ID":"9e75f3b5-4d25-4fed-91f1-99a41800f320","Type":"ContainerDied","Data":"97791a83b325a8da09ec335c1f86272ab9ae2b30adf8295bc4575915baf1a2ca"}
Oct 30 03:15:22.635943 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.635911    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb" event={"ID":"8e6f2b85-9672-4fc7-8a8a-0393e7886187","Type":"ContainerDied","Data":"f00d0c0ec3aa9b04e4f322aeb61b062e5a2406ce23c68a3186ee2d3236063e7e"}
Oct 30 03:15:22.636078 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.635962    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"
Oct 30 03:15:22.637795 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.637761    4278 scope.go:122] "RemoveContainer" containerID="1d6cde6aca877a1401435efdfd681c29cb8b8f127105331fc4914e72e58eb96d"
Oct 30 03:15:22.650170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.650153    4278 scope.go:122] "RemoveContainer" containerID="61b3c3d85a6a750a42523388cf153136a69f40f9f5b64e9ac04645a452599557"
Oct 30 03:15:22.661801 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.661776    4278 scope.go:122] "RemoveContainer" containerID="483411bd1c3f400fe52b2a1c10f3bf557361df081e29e23b3c207879992a74b6"
Oct 30 03:15:22.672070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.672046    4278 scope.go:122] "RemoveContainer" containerID="6084c1cef8c76c3ae45fb7ec7717353c0ed78164ed93cc6c72e72ccbadaf472a"
Oct 30 03:15:22.682629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.682605    4278 scope.go:122] "RemoveContainer" containerID="62f3490efc3fe72f674a7f7db2fa808f1c569f98e06352e9b073a8285bbf6d72"
Oct 30 03:15:22.686203 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.686179    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"]
Oct 30 03:15:22.698435 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.698400    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f415d-29888835-j6hws"]
Oct 30 03:15:22.732797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.732759    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"]
Oct 30 03:15:22.739129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.739092    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-537001-29888835-7sn4x"]
Oct 30 03:15:22.770056 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.770021    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"]
Oct 30 03:15:22.774256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.774226    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0b40d-29888835-447sj"]
Oct 30 03:15:22.808779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.808742    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"]
Oct 30 03:15:22.812949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.812917    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b5f19-29888835-rtbcb"]
Oct 30 03:15:22.848706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.848670    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"]
Oct 30 03:15:22.852786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.852757    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888835-cxzgb"]
Oct 30 03:15:22.885189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.885093    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"]
Oct 30 03:15:22.889173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:22.889144    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9245d1-29888835-f74ws"]
Oct 30 03:15:24.193779 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:24.193739    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:15:24.206419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:24.206389    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd" path="/var/lib/kubelet/pods/0d2f4868-04fa-430b-93bc-9644c7a526cd/volumes"
Oct 30 03:15:24.206989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:24.206972    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" path="/var/lib/kubelet/pods/8223ce14-326f-40bd-bb96-dc1b24bb4fd0/volumes"
Oct 30 03:15:24.207388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:24.207374    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" path="/var/lib/kubelet/pods/8e6f2b85-9672-4fc7-8a8a-0393e7886187/volumes"
Oct 30 03:15:24.208187 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:24.208169    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320" path="/var/lib/kubelet/pods/9e75f3b5-4d25-4fed-91f1-99a41800f320/volumes"
Oct 30 03:15:24.208636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:24.208618    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6c297a-e973-4e59-859f-5c4862900275" path="/var/lib/kubelet/pods/ea6c297a-e973-4e59-859f-5c4862900275/volumes"
Oct 30 03:15:24.209161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:24.209144    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" path="/var/lib/kubelet/pods/fed730a6-41b7-4cfc-af55-ae03dbc48d38/volumes"
Oct 30 03:15:25.651050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:25.651013    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:25.664851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:25.664823    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:25.680056 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:25.680029    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:15:28.868396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.868361    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"]
Oct 30 03:15:28.869105 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869089    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerName="cleanup"
Oct 30 03:15:28.869167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869105    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerName="cleanup"
Oct 30 03:15:28.869167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869119    4278 cpu_manager.go:414] "containerMap: removing container" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerName="cleanup"
Oct 30 03:15:28.869167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869124    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerName="cleanup"
Oct 30 03:15:28.869167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869131    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerName="cleanup"
Oct 30 03:15:28.869167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869136    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerName="cleanup"
Oct 30 03:15:28.869167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869147    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerName="cleanup"
Oct 30 03:15:28.869167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869151    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerName="cleanup"
Oct 30 03:15:28.869167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869156    4278 cpu_manager.go:414] "containerMap: removing container" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerName="cleanup"
Oct 30 03:15:28.869167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869161    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerName="cleanup"
Oct 30 03:15:28.869167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869169    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869176    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869184    4278 cpu_manager.go:414] "containerMap: removing container" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869190    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869200    4278 cpu_manager.go:414] "containerMap: removing container" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869206    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869216    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869222    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869240    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869246    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869264    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869269    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869275    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869279    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869285    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869289    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869299    4278 cpu_manager.go:414] "containerMap: removing container" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869311    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869316    4278 cpu_manager.go:414] "containerMap: removing container" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869320    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869339    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869344    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869351    4278 cpu_manager.go:414] "containerMap: removing container" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869355    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869362    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869366    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869372    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ea6c297a-e973-4e59-859f-5c4862900275" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869376    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6c297a-e973-4e59-859f-5c4862900275" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869397    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869402    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869409    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ea6c297a-e973-4e59-859f-5c4862900275" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869413    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6c297a-e973-4e59-859f-5c4862900275" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869436    4278 cpu_manager.go:414] "containerMap: removing container" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869443    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869452    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ea6c297a-e973-4e59-859f-5c4862900275" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869459    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6c297a-e973-4e59-859f-5c4862900275" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869470    4278 cpu_manager.go:414] "containerMap: removing container" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869476    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869485    4278 cpu_manager.go:414] "containerMap: removing container" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869491    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869507    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869513    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869525    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869532    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869542    4278 cpu_manager.go:414] "containerMap: removing container" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869546    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869577    4278 cpu_manager.go:414] "containerMap: removing container" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869583    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869590    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869594    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869606    4278 cpu_manager.go:414] "containerMap: removing container" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869610    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869620    4278 cpu_manager.go:414] "containerMap: removing container" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869624    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869635    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869642    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869661    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerName="cleanup"
Oct 30 03:15:28.869713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869666    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869854    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869866    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869876    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869884    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869892    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869899    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869908    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869918    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869924    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869931    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8223ce14-326f-40bd-bb96-dc1b24bb4fd0" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869941    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869949    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869956    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869965    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869972    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed730a6-41b7-4cfc-af55-ae03dbc48d38" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869980    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.869988    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6c297a-e973-4e59-859f-5c4862900275" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870001    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6c297a-e973-4e59-859f-5c4862900275" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870013    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6c297a-e973-4e59-859f-5c4862900275" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870023    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870029    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870036    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870046    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870054    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870060    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870081    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870091    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1d34d8-4d62-4a52-bb97-ae9ea24eac00" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870101    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870108    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870122    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerName="cleanup"
Oct 30 03:15:28.872453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.870129    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerName="cleanup"
Oct 30 03:15:28.874689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.874660    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:28.896129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.896068    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"]
Oct 30 03:15:28.909041 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.909005    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2305f4\"/\"kube-root-ca.crt\""
Oct 30 03:15:28.909234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.909151    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2305f4\"/\"test-model-2305f4-kserve-self-signed-certs\""
Oct 30 03:15:28.912398 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.909637    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2305f4\"/\"openshift-service-ca.crt\""
Oct 30 03:15:28.912398 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.909736    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2305f4\"/\"default-dockercfg-bqcpd\""
Oct 30 03:15:28.964310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.964268    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-model-cache\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:28.964515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.964434    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-tmp-dir\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:28.964591 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.964515    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-home\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:28.964656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.964589    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18642810-698e-4f33-a220-dd28c6b54949-tls-certs\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:28.964865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.964834    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-dshm\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:28.965002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:28.964905    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg9h5\" (UniqueName: \"kubernetes.io/projected/18642810-698e-4f33-a220-dd28c6b54949-kube-api-access-vg9h5\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.066334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.066295    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-model-cache\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.066579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.066383    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-tmp-dir\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.066579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.066457    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-home\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.066579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.066555    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18642810-698e-4f33-a220-dd28c6b54949-tls-certs\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.066744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.066726    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-dshm\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.066800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.066785    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg9h5\" (UniqueName: \"kubernetes.io/projected/18642810-698e-4f33-a220-dd28c6b54949-kube-api-access-vg9h5\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.067466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.067414    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-model-cache\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.067608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.067465    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-tmp-dir\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.067608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.067499    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-home\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.069562 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.069535    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-dshm\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.069836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.069817    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18642810-698e-4f33-a220-dd28c6b54949-tls-certs\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.104508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.100073    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg9h5\" (UniqueName: \"kubernetes.io/projected/18642810-698e-4f33-a220-dd28c6b54949-kube-api-access-vg9h5\") pod \"test-model-2305f4-kserve-574fd5b8b7-rvxxk\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") " pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.226529 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.221776    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:29.812654 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:29.812611    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"]
Oct 30 03:15:30.827712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:30.827664    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk" event={"ID":"18642810-698e-4f33-a220-dd28c6b54949","Type":"ContainerStarted","Data":"a8b76446f7f82034488543d78ac173f33dd93ad55e748dc07903fdd34bd6f95d"}
Oct 30 03:15:30.828195 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:30.827722    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk" event={"ID":"18642810-698e-4f33-a220-dd28c6b54949","Type":"ContainerStarted","Data":"50c0116065ae44d92cd422038161f4a7cea67809427f17f074b6ce9311440bf8"}
Oct 30 03:15:30.873012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:30.872953    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk" podStartSLOduration=2.872934594 podStartE2EDuration="2.872934594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:30.871201986 +0000 UTC m=+2068.795410171" watchObservedRunningTime="2026-10-30 03:15:30.872934594 +0000 UTC m=+2068.797142787"
Oct 30 03:15:31.222716 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:31.222628    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:39.222618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:39.222540    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:39.256596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:39.256563    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:39.946093 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:39.946062    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:15:44.915713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:44.915666    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:15:47.009516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:47.009468    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"]
Oct 30 03:15:47.011073 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:47.011027    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy" containerID="cri-o://a62b472d8f878f0394e45bc854e9e1a082369fe50011819638d6433c98b4aca3" gracePeriod=30
Oct 30 03:15:48.430524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:48.430482    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"]
Oct 30 03:15:48.431401 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:48.431363    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" containerID="cri-o://9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511" gracePeriod=30
Oct 30 03:15:48.444556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:48.438931    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"]
Oct 30 03:15:48.467798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:48.467750    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy" containerID="cri-o://fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38" gracePeriod=30
Oct 30 03:15:48.688094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:48.687999    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"]
Oct 30 03:15:48.688573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:48.688541    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" containerName="istio-proxy" containerID="cri-o://2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f" gracePeriod=30
Oct 30 03:15:48.700501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:48.700337    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"]
Oct 30 03:15:48.701076 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:48.701028    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy" containerID="cri-o://989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339" gracePeriod=30
Oct 30 03:15:48.735974 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:48.732762    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"]
Oct 30 03:15:48.735974 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:48.734948    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy" containerID="cri-o://4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549" gracePeriod=30
Oct 30 03:15:49.022448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:49.015005    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"]
Oct 30 03:15:49.022448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:49.015731    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy" containerID="cri-o://5290a680e48b9c1274bc19e29b56ec2086da0bb5bb7d447c0d935f907640d734" gracePeriod=30
Oct 30 03:15:51.232118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:51.232074    4278 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:15:51.232614 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:51.232156    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:15:52.052070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.052027    4278 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:15:52.052369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.052335    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:15:52.222808 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.222768    4278 generic.go:356] "Generic (PLEG): container finished" podID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerID="a62b472d8f878f0394e45bc854e9e1a082369fe50011819638d6433c98b4aca3" exitCode=0
Oct 30 03:15:52.223380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.223339    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" event={"ID":"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7","Type":"ContainerDied","Data":"a62b472d8f878f0394e45bc854e9e1a082369fe50011819638d6433c98b4aca3"}
Oct 30 03:15:52.334095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.334062    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:15:52.335307 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.335274    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:15:52.335486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.335351    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:15:52.344034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.344001    4278 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:15:52.344172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.344063    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:15:52.419127 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419087    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-credential-socket\") pod \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") "
Oct 30 03:15:52.419356 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419171    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-socket\") pod \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") "
Oct 30 03:15:52.419356 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419232    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-token\" (UniqueName: \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-token\") pod \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") "
Oct 30 03:15:52.419565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419449    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-credential-socket" pod "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" (UID: "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:52.419565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419418    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-kube-api-access-lm8xm\" (UniqueName: \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-kube-api-access-lm8xm\") pod \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") "
Oct 30 03:15:52.419565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419523    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-podinfo\") pod \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") "
Oct 30 03:15:52.419732 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419585    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istiod-ca-cert\") pod \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") "
Oct 30 03:15:52.419789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419752    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-certs\") pod \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") "
Oct 30 03:15:52.419789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419771    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-socket" pod "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" (UID: "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:52.419899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419805    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-envoy\") pod \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") "
Oct 30 03:15:52.419956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.419895    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-data\") pod \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\" (UID: \"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7\") "
Oct 30 03:15:52.420149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.420125    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-certs" pod "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" (UID: "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:52.420228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.420183    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istiod-ca-cert" pod "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" (UID: "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:52.420402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.420377    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-data" pod "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" (UID: "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:52.420947 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.420919    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.421053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.420951    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.421053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.420964    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.421053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.420978    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.421053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.420992    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.422981 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.422954    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-token" pod "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" (UID: "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:52.423092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.422989    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-kube-api-access-lm8xm" pod "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" (UID: "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7"). InnerVolumeSpecName "kube-api-access-lm8xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:52.423092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.422999    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-envoy" pod "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" (UID: "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:52.423448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.423402    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-podinfo" pod "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" (UID: "48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:52.523058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.523022    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.523058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.523059    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8xm\" (UniqueName: \"kubernetes.io/projected/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-kube-api-access-lm8xm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.523305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.523075    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.523305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:52.523090    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:53.029167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:53.029128    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"]
Oct 30 03:15:53.041726 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:53.041695    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1991a36-92t7v"]
Oct 30 03:15:53.234085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:53.234053    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"
Oct 30 03:15:53.234085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:53.234072    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79" event={"ID":"48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7","Type":"ContainerDied","Data":"82c64185cd8901c312c4cf242aa2b604db4176cc8dc4f14a6c30551f84240142"}
Oct 30 03:15:53.234368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:53.234148    4278 scope.go:122] "RemoveContainer" containerID="a62b472d8f878f0394e45bc854e9e1a082369fe50011819638d6433c98b4aca3"
Oct 30 03:15:53.290175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:53.290095    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"]
Oct 30 03:15:53.302128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:53.302097    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kdl79"]
Oct 30 03:15:53.994083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:53.994051    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 03:15:54.005472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.002628    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 03:15:54.082639 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.082477    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-envoy\") pod \"b096fb20-7117-439d-8901-7e9e9321c044\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") "
Oct 30 03:15:54.082885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.082718    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b096fb20-7117-439d-8901-7e9e9321c044-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096fb20-7117-439d-8901-7e9e9321c044-istiod-ca-cert\") pod \"b096fb20-7117-439d-8901-7e9e9321c044\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") "
Oct 30 03:15:54.082885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.082769    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-kube-api-access-g7jnn\" (UniqueName: \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-kube-api-access-g7jnn\") pod \"b096fb20-7117-439d-8901-7e9e9321c044\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") "
Oct 30 03:15:54.082885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.082847    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-data\") pod \"b096fb20-7117-439d-8901-7e9e9321c044\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") "
Oct 30 03:15:54.083083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.082962    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b096fb20-7117-439d-8901-7e9e9321c044-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096fb20-7117-439d-8901-7e9e9321c044-istio-podinfo\") pod \"b096fb20-7117-439d-8901-7e9e9321c044\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") "
Oct 30 03:15:54.083083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.083019    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-credential-socket\") pod \"b096fb20-7117-439d-8901-7e9e9321c044\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") "
Oct 30 03:15:54.084233 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.083313    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-certs\") pod \"b096fb20-7117-439d-8901-7e9e9321c044\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") "
Oct 30 03:15:54.084233 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.083410    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-socket\") pod \"b096fb20-7117-439d-8901-7e9e9321c044\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") "
Oct 30 03:15:54.084233 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.083572    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-istio-token\" (UniqueName: \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-istio-token\") pod \"b096fb20-7117-439d-8901-7e9e9321c044\" (UID: \"b096fb20-7117-439d-8901-7e9e9321c044\") "
Oct 30 03:15:54.094897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.092051    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-credential-socket" pod "b096fb20-7117-439d-8901-7e9e9321c044" (UID: "b096fb20-7117-439d-8901-7e9e9321c044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.094897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.092227    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-data" pod "b096fb20-7117-439d-8901-7e9e9321c044" (UID: "b096fb20-7117-439d-8901-7e9e9321c044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.094897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.092407    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-certs" pod "b096fb20-7117-439d-8901-7e9e9321c044" (UID: "b096fb20-7117-439d-8901-7e9e9321c044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.094897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.092622    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b096fb20-7117-439d-8901-7e9e9321c044-istiod-ca-cert" pod "b096fb20-7117-439d-8901-7e9e9321c044" (UID: "b096fb20-7117-439d-8901-7e9e9321c044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:54.097483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.095367    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-socket" pod "b096fb20-7117-439d-8901-7e9e9321c044" (UID: "b096fb20-7117-439d-8901-7e9e9321c044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.099937 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.099874    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-kube-api-access-g7jnn" pod "b096fb20-7117-439d-8901-7e9e9321c044" (UID: "b096fb20-7117-439d-8901-7e9e9321c044"). InnerVolumeSpecName "kube-api-access-g7jnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.099937 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.099912    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-istio-token" pod "b096fb20-7117-439d-8901-7e9e9321c044" (UID: "b096fb20-7117-439d-8901-7e9e9321c044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.100573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.100532    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-envoy" pod "b096fb20-7117-439d-8901-7e9e9321c044" (UID: "b096fb20-7117-439d-8901-7e9e9321c044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.102553 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.102521    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b096fb20-7117-439d-8901-7e9e9321c044-istio-podinfo" pod "b096fb20-7117-439d-8901-7e9e9321c044" (UID: "b096fb20-7117-439d-8901-7e9e9321c044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:54.139206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.139173    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:15:54.168008 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.167973    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 03:15:54.186058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.186023    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 03:15:54.186708 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.186678    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-credential-socket\") pod \"6fbe77ca-1cd6-440f-a919-921fb337fca1\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") "
Oct 30 03:15:54.186830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.186787    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-envoy\") pod \"6fbe77ca-1cd6-440f-a919-921fb337fca1\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") "
Oct 30 03:15:54.186894 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.186837    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-podinfo\") pod \"6fbe77ca-1cd6-440f-a919-921fb337fca1\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") "
Oct 30 03:15:54.186961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.186914    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-certs\") pod \"6fbe77ca-1cd6-440f-a919-921fb337fca1\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") "
Oct 30 03:15:54.187153 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.187135    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-socket\") pod \"6fbe77ca-1cd6-440f-a919-921fb337fca1\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") "
Oct 30 03:15:54.187241 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.187223    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-kube-api-access-jdz92\" (UniqueName: \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-kube-api-access-jdz92\") pod \"6fbe77ca-1cd6-440f-a919-921fb337fca1\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") "
Oct 30 03:15:54.187307 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.187276    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fbe77ca-1cd6-440f-a919-921fb337fca1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fbe77ca-1cd6-440f-a919-921fb337fca1-istiod-ca-cert\") pod \"6fbe77ca-1cd6-440f-a919-921fb337fca1\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") "
Oct 30 03:15:54.187376 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.187361    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-data\") pod \"6fbe77ca-1cd6-440f-a919-921fb337fca1\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") "
Oct 30 03:15:54.187545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.187409    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-token\" (UniqueName: \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-token\") pod \"6fbe77ca-1cd6-440f-a919-921fb337fca1\" (UID: \"6fbe77ca-1cd6-440f-a919-921fb337fca1\") "
Oct 30 03:15:54.188401 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.188367    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.188401 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.188399    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.188657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.188413    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.188657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.188469    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096fb20-7117-439d-8901-7e9e9321c044-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.188657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.188483    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7jnn\" (UniqueName: \"kubernetes.io/projected/b096fb20-7117-439d-8901-7e9e9321c044-kube-api-access-g7jnn\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.188657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.188496    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.188657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.188510    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096fb20-7117-439d-8901-7e9e9321c044-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.188657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.188523    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.188657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.188535    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096fb20-7117-439d-8901-7e9e9321c044-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.189252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.189217    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-socket" pod "6fbe77ca-1cd6-440f-a919-921fb337fca1" (UID: "6fbe77ca-1cd6-440f-a919-921fb337fca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.189637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.189543    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-data" pod "6fbe77ca-1cd6-440f-a919-921fb337fca1" (UID: "6fbe77ca-1cd6-440f-a919-921fb337fca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.190091 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.190016    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbe77ca-1cd6-440f-a919-921fb337fca1-istiod-ca-cert" pod "6fbe77ca-1cd6-440f-a919-921fb337fca1" (UID: "6fbe77ca-1cd6-440f-a919-921fb337fca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:54.190164 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.190118    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-credential-socket" pod "6fbe77ca-1cd6-440f-a919-921fb337fca1" (UID: "6fbe77ca-1cd6-440f-a919-921fb337fca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.190909 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.190868    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-certs" pod "6fbe77ca-1cd6-440f-a919-921fb337fca1" (UID: "6fbe77ca-1cd6-440f-a919-921fb337fca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.192596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.192575    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-token" pod "6fbe77ca-1cd6-440f-a919-921fb337fca1" (UID: "6fbe77ca-1cd6-440f-a919-921fb337fca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.192783 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.192764    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-kube-api-access-jdz92" pod "6fbe77ca-1cd6-440f-a919-921fb337fca1" (UID: "6fbe77ca-1cd6-440f-a919-921fb337fca1"). InnerVolumeSpecName "kube-api-access-jdz92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.195585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.195561    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-podinfo" pod "6fbe77ca-1cd6-440f-a919-921fb337fca1" (UID: "6fbe77ca-1cd6-440f-a919-921fb337fca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:54.195705 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.195580    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-envoy" pod "6fbe77ca-1cd6-440f-a919-921fb337fca1" (UID: "6fbe77ca-1cd6-440f-a919-921fb337fca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.213584 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.213018    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0242ff22-6f8a-4f8f-a820-5e6ab21378ef" path="/var/lib/kubelet/pods/0242ff22-6f8a-4f8f-a820-5e6ab21378ef/volumes"
Oct 30 03:15:54.214861 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.213978    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" path="/var/lib/kubelet/pods/48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7/volumes"
Oct 30 03:15:54.216561 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:54.215796    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:15:54.271383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.270814    4278 generic.go:356] "Generic (PLEG): container finished" podID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerID="5290a680e48b9c1274bc19e29b56ec2086da0bb5bb7d447c0d935f907640d734" exitCode=0
Oct 30 03:15:54.271383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.270929    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" event={"ID":"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0","Type":"ContainerDied","Data":"5290a680e48b9c1274bc19e29b56ec2086da0bb5bb7d447c0d935f907640d734"}
Oct 30 03:15:54.273157 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.273126    4278 generic.go:356] "Generic (PLEG): container finished" podID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerID="4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549" exitCode=0
Oct 30 03:15:54.273398 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.273377    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" event={"ID":"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf","Type":"ContainerDied","Data":"4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549"}
Oct 30 03:15:54.273580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.273562    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht" event={"ID":"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf","Type":"ContainerDied","Data":"b4393e691f1a032a806bcc26a70c24ec6f6d24fb00fc816285ae31d26b4f499f"}
Oct 30 03:15:54.273705 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.273692    4278 scope.go:122] "RemoveContainer" containerID="4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549"
Oct 30 03:15:54.273976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.273958    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"
Oct 30 03:15:54.281225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.281198    4278 generic.go:356] "Generic (PLEG): container finished" podID="b096fb20-7117-439d-8901-7e9e9321c044" containerID="9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511" exitCode=0
Oct 30 03:15:54.281387 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.281371    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"
Oct 30 03:15:54.282160 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.282127    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" event={"ID":"b096fb20-7117-439d-8901-7e9e9321c044","Type":"ContainerDied","Data":"9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511"}
Oct 30 03:15:54.282272 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.282190    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5" event={"ID":"b096fb20-7117-439d-8901-7e9e9321c044","Type":"ContainerDied","Data":"03122a68df48003a2b7f52272d3b4cb573089d575f63364e13c2c1048979bbe9"}
Oct 30 03:15:54.285928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.283986    4278 generic.go:356] "Generic (PLEG): container finished" podID="0b4bc478-f00d-435b-be38-c858acdc162c" containerID="2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f" exitCode=0
Oct 30 03:15:54.285928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.284180    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"
Oct 30 03:15:54.285928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.284259    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" event={"ID":"0b4bc478-f00d-435b-be38-c858acdc162c","Type":"ContainerDied","Data":"2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f"}
Oct 30 03:15:54.285928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.284312    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx" event={"ID":"0b4bc478-f00d-435b-be38-c858acdc162c","Type":"ContainerDied","Data":"ca53f1083812e88e44edd34be42df742dbe7cdb6ee427d7f571738e81b4f0421"}
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.294645    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-kube-api-access-976hn\" (UniqueName: \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-kube-api-access-976hn\") pod \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.294743    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" event={"ID":"6fbe77ca-1cd6-440f-a919-921fb337fca1","Type":"ContainerDied","Data":"fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38"}
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.294758    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-credential-socket\") pod \"b2a68760-ccdd-4222-9463-862df0c25ed3\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.294655    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.294819    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-credential-socket\") pod \"0b4bc478-f00d-435b-be38-c858acdc162c\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.294873    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-podinfo\") pod \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.295628    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-envoy\") pod \"b2a68760-ccdd-4222-9463-862df0c25ed3\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.295701    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-certs\") pod \"0b4bc478-f00d-435b-be38-c858acdc162c\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.295741    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4bc478-f00d-435b-be38-c858acdc162c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4bc478-f00d-435b-be38-c858acdc162c-istiod-ca-cert\") pod \"0b4bc478-f00d-435b-be38-c858acdc162c\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.295773    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b4bc478-f00d-435b-be38-c858acdc162c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4bc478-f00d-435b-be38-c858acdc162c-istio-podinfo\") pod \"0b4bc478-f00d-435b-be38-c858acdc162c\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.295809    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-credential-socket\") pod \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.295901    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-podinfo\") pod \"b2a68760-ccdd-4222-9463-862df0c25ed3\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.295947    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-istio-token\" (UniqueName: \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-istio-token\") pod \"0b4bc478-f00d-435b-be38-c858acdc162c\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.295980    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-envoy\") pod \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296020    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-socket\") pod \"b2a68760-ccdd-4222-9463-862df0c25ed3\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296110    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-socket\") pod \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296161    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-socket\") pod \"0b4bc478-f00d-435b-be38-c858acdc162c\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296204    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istiod-ca-cert\") pod \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296239    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-data\") pod \"0b4bc478-f00d-435b-be38-c858acdc162c\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296279    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a68760-ccdd-4222-9463-862df0c25ed3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a68760-ccdd-4222-9463-862df0c25ed3-istiod-ca-cert\") pod \"b2a68760-ccdd-4222-9463-862df0c25ed3\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296329    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-data\") pod \"b2a68760-ccdd-4222-9463-862df0c25ed3\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296374    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-envoy\") pod \"0b4bc478-f00d-435b-be38-c858acdc162c\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296438    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-token\" (UniqueName: \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-token\") pod \"b2a68760-ccdd-4222-9463-862df0c25ed3\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") "
Oct 30 03:15:54.296463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296480    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-kube-api-access-nf4pm\" (UniqueName: \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-kube-api-access-nf4pm\") pod \"0b4bc478-f00d-435b-be38-c858acdc162c\" (UID: \"0b4bc478-f00d-435b-be38-c858acdc162c\") "
Oct 30 03:15:54.297758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296639    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-certs\") pod \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") "
Oct 30 03:15:54.297758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.296696    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-token\") pod \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") "
Oct 30 03:15:54.297758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.297171    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-credential-socket" pod "b2a68760-ccdd-4222-9463-862df0c25ed3" (UID: "b2a68760-ccdd-4222-9463-862df0c25ed3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.297758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.297611    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-kube-api-access-w42xl\" (UniqueName: \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-kube-api-access-w42xl\") pod \"b2a68760-ccdd-4222-9463-862df0c25ed3\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") "
Oct 30 03:15:54.297758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.297673    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-data\") pod \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\" (UID: \"cb0ec525-1d16-4d6c-ab56-19a341bbc3cf\") "
Oct 30 03:15:54.297758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.297723    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-certs\") pod \"b2a68760-ccdd-4222-9463-862df0c25ed3\" (UID: \"b2a68760-ccdd-4222-9463-862df0c25ed3\") "
Oct 30 03:15:54.298831 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.298070    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-socket" pod "b2a68760-ccdd-4222-9463-862df0c25ed3" (UID: "b2a68760-ccdd-4222-9463-862df0c25ed3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.298831 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.298331    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-credential-socket" pod "0b4bc478-f00d-435b-be38-c858acdc162c" (UID: "0b4bc478-f00d-435b-be38-c858acdc162c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.300172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.300151    4278 scope.go:122] "RemoveContainer" containerID="4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549"
Oct 30 03:15:54.301489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.294410    4278 generic.go:356] "Generic (PLEG): container finished" podID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerID="fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38" exitCode=0
Oct 30 03:15:54.302467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.301676    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-podinfo" pod "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" (UID: "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:54.302594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.302019    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-credential-socket" pod "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" (UID: "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.302757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.302494    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c" event={"ID":"6fbe77ca-1cd6-440f-a919-921fb337fca1","Type":"ContainerDied","Data":"2a13f7e57688b75eb7528a15cc9cdf7750a90aff431aa76680a370dbe95aefe5"}
Oct 30 03:15:54.302863 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:54.302786    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549\": container with ID starting with 4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549 not found: ID does not exist" containerID="4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549"
Oct 30 03:15:54.302863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.302824    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549"} err="failed to get container status \"4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549\": rpc error: code = NotFound desc = could not find container \"4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549\": container with ID starting with 4d2d2422eeb3b87acb19000196480d0b47cd267d9cfbca0231d1951751d81549 not found: ID does not exist"
Oct 30 03:15:54.302863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.302847    4278 scope.go:122] "RemoveContainer" containerID="9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511"
Oct 30 03:15:54.303250 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.303225    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-certs" pod "0b4bc478-f00d-435b-be38-c858acdc162c" (UID: "0b4bc478-f00d-435b-be38-c858acdc162c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.304293 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.304272    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4bc478-f00d-435b-be38-c858acdc162c-istiod-ca-cert" pod "0b4bc478-f00d-435b-be38-c858acdc162c" (UID: "0b4bc478-f00d-435b-be38-c858acdc162c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:54.305093 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.305049    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-socket" pod "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" (UID: "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.305268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.305246    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-socket" pod "0b4bc478-f00d-435b-be38-c858acdc162c" (UID: "0b4bc478-f00d-435b-be38-c858acdc162c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.305360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.305260    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b4bc478-f00d-435b-be38-c858acdc162c-istio-podinfo" pod "0b4bc478-f00d-435b-be38-c858acdc162c" (UID: "0b4bc478-f00d-435b-be38-c858acdc162c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.305661    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istiod-ca-cert" pod "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" (UID: "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.305902    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-data" pod "0b4bc478-f00d-435b-be38-c858acdc162c" (UID: "0b4bc478-f00d-435b-be38-c858acdc162c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.306253    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a68760-ccdd-4222-9463-862df0c25ed3-istiod-ca-cert" pod "b2a68760-ccdd-4222-9463-862df0c25ed3" (UID: "b2a68760-ccdd-4222-9463-862df0c25ed3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.306508    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-data" pod "b2a68760-ccdd-4222-9463-862df0c25ed3" (UID: "b2a68760-ccdd-4222-9463-862df0c25ed3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.306813    4278 generic.go:356] "Generic (PLEG): container finished" podID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerID="989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339" exitCode=0
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.306868    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" event={"ID":"b2a68760-ccdd-4222-9463-862df0c25ed3","Type":"ContainerDied","Data":"989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339"}
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.306918    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" event={"ID":"b2a68760-ccdd-4222-9463-862df0c25ed3","Type":"ContainerDied","Data":"0ce9eca7621f295efdfe9149752f613df191c00ece3b7c3bd6b517728ab32f99"}
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.307018    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.307074    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-podinfo" pod "b2a68760-ccdd-4222-9463-862df0c25ed3" (UID: "b2a68760-ccdd-4222-9463-862df0c25ed3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.307295    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-certs" pod "b2a68760-ccdd-4222-9463-862df0c25ed3" (UID: "b2a68760-ccdd-4222-9463-862df0c25ed3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.307547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.307308    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-data" pod "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" (UID: "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.308131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308087    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308115    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308130    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308143    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308161    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308176    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308189    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308203    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4bc478-f00d-435b-be38-c858acdc162c-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308215    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4bc478-f00d-435b-be38-c858acdc162c-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308227    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308240    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308253    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308267    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308281    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308295    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308309    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308323    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.308340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308337    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a68760-ccdd-4222-9463-862df0c25ed3-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308351    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308368    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdz92\" (UniqueName: \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-kube-api-access-jdz92\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308383    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fbe77ca-1cd6-440f-a919-921fb337fca1-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308398    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308411    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fbe77ca-1cd6-440f-a919-921fb337fca1-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308443    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308473    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308486    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbe77ca-1cd6-440f-a919-921fb337fca1-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308619    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-certs" pod "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" (UID: "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.308818    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-envoy" pod "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" (UID: "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.309001    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-kube-api-access-976hn" pod "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" (UID: "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf"). InnerVolumeSpecName "kube-api-access-976hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.309066    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-envoy" pod "0b4bc478-f00d-435b-be38-c858acdc162c" (UID: "0b4bc478-f00d-435b-be38-c858acdc162c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.309261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.309256    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-token" pod "b2a68760-ccdd-4222-9463-862df0c25ed3" (UID: "b2a68760-ccdd-4222-9463-862df0c25ed3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.309896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.309663    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-istio-token" pod "0b4bc478-f00d-435b-be38-c858acdc162c" (UID: "0b4bc478-f00d-435b-be38-c858acdc162c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.311367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.311337    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-envoy" pod "b2a68760-ccdd-4222-9463-862df0c25ed3" (UID: "b2a68760-ccdd-4222-9463-862df0c25ed3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.311496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.311367    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-token" pod "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" (UID: "cb0ec525-1d16-4d6c-ab56-19a341bbc3cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.312040 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.312012    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-kube-api-access-w42xl" pod "b2a68760-ccdd-4222-9463-862df0c25ed3" (UID: "b2a68760-ccdd-4222-9463-862df0c25ed3"). InnerVolumeSpecName "kube-api-access-w42xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.312581 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.312556    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-kube-api-access-nf4pm" pod "0b4bc478-f00d-435b-be38-c858acdc162c" (UID: "0b4bc478-f00d-435b-be38-c858acdc162c"). InnerVolumeSpecName "kube-api-access-nf4pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.320573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.320395    4278 scope.go:122] "RemoveContainer" containerID="9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511"
Oct 30 03:15:54.321051 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:54.320891    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511\": container with ID starting with 9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511 not found: ID does not exist" containerID="9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511"
Oct 30 03:15:54.321051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.320933    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511"} err="failed to get container status \"9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511\": rpc error: code = NotFound desc = could not find container \"9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511\": container with ID starting with 9edb91cf90d6fecc085d74e50e686448933f705056dcb6d5d3ef32631a5d5511 not found: ID does not exist"
Oct 30 03:15:54.321051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.320956    4278 scope.go:122] "RemoveContainer" containerID="2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f"
Oct 30 03:15:54.337022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.336998    4278 scope.go:122] "RemoveContainer" containerID="2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f"
Oct 30 03:15:54.337529 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:54.337367    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f\": container with ID starting with 2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f not found: ID does not exist" containerID="2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f"
Oct 30 03:15:54.337529 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.337410    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f"} err="failed to get container status \"2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f\": rpc error: code = NotFound desc = could not find container \"2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f\": container with ID starting with 2b64c878850e100a8f23e2fbb35b533e58ab8a550fe5fa6ae8897926d70fb07f not found: ID does not exist"
Oct 30 03:15:54.337529 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.337453    4278 scope.go:122] "RemoveContainer" containerID="fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38"
Oct 30 03:15:54.369766 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.369739    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 03:15:54.372633 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.372601    4278 scope.go:122] "RemoveContainer" containerID="fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38"
Oct 30 03:15:54.372986 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:54.372956    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38\": container with ID starting with fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38 not found: ID does not exist" containerID="fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38"
Oct 30 03:15:54.373069 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.372998    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38"} err="failed to get container status \"fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38\": rpc error: code = NotFound desc = could not find container \"fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38\": container with ID starting with fb5ba80fd6084a84ae2d18e8e62ad3e307ed6d64b144da73c8dec1021a742b38 not found: ID does not exist"
Oct 30 03:15:54.373069 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.373027    4278 scope.go:122] "RemoveContainer" containerID="989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339"
Oct 30 03:15:54.387852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.387830    4278 scope.go:122] "RemoveContainer" containerID="989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339"
Oct 30 03:15:54.388220 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:15:54.388188    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339\": container with ID starting with 989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339 not found: ID does not exist" containerID="989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339"
Oct 30 03:15:54.388304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.388231    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339"} err="failed to get container status \"989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339\": rpc error: code = NotFound desc = could not find container \"989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339\": container with ID starting with 989a6403f128a2ddbf77d9b7084c15f944b62a0eed9c1a60cae35e3164bfd339 not found: ID does not exist"
Oct 30 03:15:54.409537 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.409507    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-envoy\") pod \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") "
Oct 30 03:15:54.409648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.409629    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-credential-socket\") pod \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") "
Oct 30 03:15:54.409699 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.409687    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-token\" (UniqueName: \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-token\") pod \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") "
Oct 30 03:15:54.409737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.409724    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-certs\") pod \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") "
Oct 30 03:15:54.409769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.409751    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-data\") pod \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") "
Oct 30 03:15:54.409803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.409787    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-podinfo\") pod \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") "
Oct 30 03:15:54.409917 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.409893    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-kube-api-access-gqgkc\" (UniqueName: \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-kube-api-access-gqgkc\") pod \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") "
Oct 30 03:15:54.409999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.409968    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-credential-socket" pod "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" (UID: "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.410055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.410039    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istiod-ca-cert\") pod \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") "
Oct 30 03:15:54.410112 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.410064    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-data" pod "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" (UID: "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.410112 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.410091    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-certs" pod "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" (UID: "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.410220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.410183    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-socket\") pod \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\" (UID: \"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0\") "
Oct 30 03:15:54.410534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.410490    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-socket" pod "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" (UID: "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.410534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.410522    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istiod-ca-cert" pod "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" (UID: "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:54.411142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411117    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411149    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411166    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411182    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411196    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411208    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411221    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4bc478-f00d-435b-be38-c858acdc162c-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411234    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411247    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf4pm\" (UniqueName: \"kubernetes.io/projected/0b4bc478-f00d-435b-be38-c858acdc162c-kube-api-access-nf4pm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411261    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411275    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411289    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411302    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411315    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w42xl\" (UniqueName: \"kubernetes.io/projected/b2a68760-ccdd-4222-9463-862df0c25ed3-kube-api-access-w42xl\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.411322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.411329    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-976hn\" (UniqueName: \"kubernetes.io/projected/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf-kube-api-access-976hn\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.412196 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.412173    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-envoy" pod "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" (UID: "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.412300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.412204    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-podinfo" pod "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" (UID: "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:54.412300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.412260    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-token" pod "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" (UID: "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.412385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.412365    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-kube-api-access-gqgkc" pod "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" (UID: "a03a29e0-ae09-4eb5-aab7-0913ab7e00b0"). InnerVolumeSpecName "kube-api-access-gqgkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.423745 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.423704    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"]
Oct 30 03:15:54.427752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.427719    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-lmhb5"]
Oct 30 03:15:54.471889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.471848    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"]
Oct 30 03:15:54.476032 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.476003    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zdp9c"]
Oct 30 03:15:54.513051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.513017    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.513051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.513050    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.513051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.513061    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.513288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.513070    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgkc\" (UniqueName: \"kubernetes.io/projected/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0-kube-api-access-gqgkc\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.647845 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.647811    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"]
Oct 30 03:15:54.653896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.653867    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-vg7ht"]
Oct 30 03:15:54.695090 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.695052    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"]
Oct 30 03:15:54.698995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.698958    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4"]
Oct 30 03:15:54.732244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.732205    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"]
Oct 30 03:15:54.740228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:54.740197    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2p5tx"]
Oct 30 03:15:55.056469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:55.056408    4278 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:15:55.056907 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:55.056517    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-2vkg4" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:55.320960 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:55.320863    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"
Oct 30 03:15:55.320960 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:55.320871    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs" event={"ID":"a03a29e0-ae09-4eb5-aab7-0913ab7e00b0","Type":"ContainerDied","Data":"ea7c819fe1ade3d47f6a0118fcb53dfbb9af1e1747f6a3a6928212a8f7f68e35"}
Oct 30 03:15:55.320960 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:55.320941    4278 scope.go:122] "RemoveContainer" containerID="5290a680e48b9c1274bc19e29b56ec2086da0bb5bb7d447c0d935f907640d734"
Oct 30 03:15:55.380495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:55.380459    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"]
Oct 30 03:15:55.393127 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:55.393092    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-qkkjs"]
Oct 30 03:15:56.202146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:56.202108    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" path="/var/lib/kubelet/pods/0b4bc478-f00d-435b-be38-c858acdc162c/volumes"
Oct 30 03:15:56.202703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:56.202681    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" path="/var/lib/kubelet/pods/6fbe77ca-1cd6-440f-a919-921fb337fca1/volumes"
Oct 30 03:15:56.203170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:56.203156    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" path="/var/lib/kubelet/pods/a03a29e0-ae09-4eb5-aab7-0913ab7e00b0/volumes"
Oct 30 03:15:56.204021 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:56.203999    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b096fb20-7117-439d-8901-7e9e9321c044" path="/var/lib/kubelet/pods/b096fb20-7117-439d-8901-7e9e9321c044/volumes"
Oct 30 03:15:56.204495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:56.204481    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" path="/var/lib/kubelet/pods/b2a68760-ccdd-4222-9463-862df0c25ed3/volumes"
Oct 30 03:15:56.205677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:15:56.205647    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" path="/var/lib/kubelet/pods/cb0ec525-1d16-4d6c-ab56-19a341bbc3cf/volumes"
Oct 30 03:16:03.448149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:03.448110    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"]
Oct 30 03:16:03.448692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:03.448372    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy" containerID="cri-o://6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e" gracePeriod=30
Oct 30 03:16:03.795773 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:03.795740    4278 scope.go:122] "RemoveContainer" containerID="81741fb535a3fbe8456fc3f58df99697d7de9db3130061a7bb153558d8969328"
Oct 30 03:16:03.806766 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:03.806740    4278 scope.go:122] "RemoveContainer" containerID="49caa015ba276cf880e5a65b1b7baa9b5e5f9fad0b019f8d0b32170a83092b3c"
Oct 30 03:16:08.514648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.514610    4278 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 03:16:08.515072 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.514668    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 03:16:08.799910 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.799875    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:16:08.952160 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952120    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-socket\") pod \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") "
Oct 30 03:16:08.952360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952247    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-token\" (UniqueName: \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-token\") pod \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") "
Oct 30 03:16:08.952360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952317    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-data\") pod \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") "
Oct 30 03:16:08.952360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952351    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-credential-socket\") pod \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") "
Oct 30 03:16:08.952590 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952462    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-socket" pod "e2163c7f-a4d6-4648-9109-2c2cffa4ec37" (UID: "e2163c7f-a4d6-4648-9109-2c2cffa4ec37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:08.952590 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952567    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-podinfo\") pod \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") "
Oct 30 03:16:08.952706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952608    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-envoy\") pod \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") "
Oct 30 03:16:08.952706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952689    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-data" pod "e2163c7f-a4d6-4648-9109-2c2cffa4ec37" (UID: "e2163c7f-a4d6-4648-9109-2c2cffa4ec37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:08.952812 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952709    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-credential-socket" pod "e2163c7f-a4d6-4648-9109-2c2cffa4ec37" (UID: "e2163c7f-a4d6-4648-9109-2c2cffa4ec37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:08.952812 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952762    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-certs\") pod \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") "
Oct 30 03:16:08.952938 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952859    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istiod-ca-cert\") pod \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") "
Oct 30 03:16:08.952938 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.952912    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-kube-api-access-t2mrn\" (UniqueName: \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-kube-api-access-t2mrn\") pod \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\" (UID: \"e2163c7f-a4d6-4648-9109-2c2cffa4ec37\") "
Oct 30 03:16:08.953081 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.953044    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-certs" pod "e2163c7f-a4d6-4648-9109-2c2cffa4ec37" (UID: "e2163c7f-a4d6-4648-9109-2c2cffa4ec37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:08.953295 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.953269    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istiod-ca-cert" pod "e2163c7f-a4d6-4648-9109-2c2cffa4ec37" (UID: "e2163c7f-a4d6-4648-9109-2c2cffa4ec37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:08.953845 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.953816    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:08.953845 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.953847    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:08.954034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.953861    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:08.954034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.953875    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:08.954034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.953889    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:08.954935 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.954903    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-podinfo" pod "e2163c7f-a4d6-4648-9109-2c2cffa4ec37" (UID: "e2163c7f-a4d6-4648-9109-2c2cffa4ec37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:16:08.955047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.954907    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-envoy" pod "e2163c7f-a4d6-4648-9109-2c2cffa4ec37" (UID: "e2163c7f-a4d6-4648-9109-2c2cffa4ec37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:08.955047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.954907    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-token" pod "e2163c7f-a4d6-4648-9109-2c2cffa4ec37" (UID: "e2163c7f-a4d6-4648-9109-2c2cffa4ec37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:08.955207 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:08.955188    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-kube-api-access-t2mrn" pod "e2163c7f-a4d6-4648-9109-2c2cffa4ec37" (UID: "e2163c7f-a4d6-4648-9109-2c2cffa4ec37"). InnerVolumeSpecName "kube-api-access-t2mrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:09.055532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.055421    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:09.055532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.055475    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:09.055532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.055486    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2mrn\" (UniqueName: \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-kube-api-access-t2mrn\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:09.055532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.055495    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2163c7f-a4d6-4648-9109-2c2cffa4ec37-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:09.441323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.441222    4278 generic.go:356] "Generic (PLEG): container finished" podID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerID="6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e" exitCode=0
Oct 30 03:16:09.441323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.441274    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"
Oct 30 03:16:09.441323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.441306    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" event={"ID":"e2163c7f-a4d6-4648-9109-2c2cffa4ec37","Type":"ContainerDied","Data":"6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e"}
Oct 30 03:16:09.441602 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.441364    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm" event={"ID":"e2163c7f-a4d6-4648-9109-2c2cffa4ec37","Type":"ContainerDied","Data":"faa3892e27d37e5a53eed27d6ca25d8c8f359863fade2e8f279cf433b3ca88bb"}
Oct 30 03:16:09.441602 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.441390    4278 scope.go:122] "RemoveContainer" containerID="6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e"
Oct 30 03:16:09.454043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.454021    4278 scope.go:122] "RemoveContainer" containerID="6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e"
Oct 30 03:16:09.454333 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:16:09.454313    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e\": container with ID starting with 6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e not found: ID does not exist" containerID="6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e"
Oct 30 03:16:09.454392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.454342    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e"} err="failed to get container status \"6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e\": rpc error: code = NotFound desc = could not find container \"6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e\": container with ID starting with 6c32ffa5746baae0f38f39f20fc3dc2710785830e658265f898ffa832d045e7e not found: ID does not exist"
Oct 30 03:16:09.509567 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.509535    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"]
Oct 30 03:16:09.532752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:09.524295    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-5btfm"]
Oct 30 03:16:10.219845 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:10.219805    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" path="/var/lib/kubelet/pods/e2163c7f-a4d6-4648-9109-2c2cffa4ec37/volumes"
Oct 30 03:16:13.053062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053013    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"]
Oct 30 03:16:13.053838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053815    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerName="cleanup"
Oct 30 03:16:13.053895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053837    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerName="cleanup"
Oct 30 03:16:13.053895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053863    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy"
Oct 30 03:16:13.053895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053868    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy"
Oct 30 03:16:13.053895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053881    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerName="cleanup"
Oct 30 03:16:13.053895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053889    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerName="cleanup"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053909    4278 cpu_manager.go:414] "containerMap: removing container" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053914    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053924    4278 cpu_manager.go:414] "containerMap: removing container" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053929    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053941    4278 cpu_manager.go:414] "containerMap: removing container" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053948    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053958    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerName="cleanup"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053965    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerName="cleanup"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053985    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerName="cleanup"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.053991    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5103b31a-7eaf-4364-9af6-1b7eb34e1d40" containerName="cleanup"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054011    4278 cpu_manager.go:414] "containerMap: removing container" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054017    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054027    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054034    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054046    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054052    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054059    4278 cpu_manager.go:414] "containerMap: removing container" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054066    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054077    4278 cpu_manager.go:414] "containerMap: removing container" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerName="cleanup"
Oct 30 03:16:13.054121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054084    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerName="cleanup"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054239    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e75f3b5-4d25-4fed-91f1-99a41800f320" containerName="cleanup"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054253    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02df413-cc1b-4d7e-8209-16022147c7a1" containerName="cleanup"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054261    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="b096fb20-7117-439d-8901-7e9e9321c044" containerName="istio-proxy"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054268    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4bc478-f00d-435b-be38-c858acdc162c" containerName="istio-proxy"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054273    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fbe77ca-1cd6-440f-a919-921fb337fca1" containerName="istio-proxy"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054279    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0ec525-1d16-4d6c-ab56-19a341bbc3cf" containerName="istio-proxy"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054283    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="969c6501-88c9-4285-9b60-2d38d788bbd6" containerName="cleanup"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054290    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfefef3e-400b-48d3-b4f8-bf432ee80c29" containerName="cleanup"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054295    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6f2b85-9672-4fc7-8a8a-0393e7886187" containerName="cleanup"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054300    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2f4868-04fa-430b-93bc-9644c7a526cd" containerName="cleanup"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054307    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c90ec76-2fe7-40e2-b88b-8b766b28e158" containerName="cleanup"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054315    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03a29e0-ae09-4eb5-aab7-0913ab7e00b0" containerName="istio-proxy"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054323    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5968a2a-86ba-4717-8f0f-df111b79f123" containerName="cleanup"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054331    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a68760-ccdd-4222-9463-862df0c25ed3" containerName="istio-proxy"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054338    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a4aa09-1b32-4ae5-80e3-43bc0a2ab6b7" containerName="istio-proxy"
Oct 30 03:16:13.055145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.054344    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2163c7f-a4d6-4648-9109-2c2cffa4ec37" containerName="istio-proxy"
Oct 30 03:16:13.059837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.059816    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.076183 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.076158    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f21058-openshift-default-dockercfg-wb4zh\""
Oct 30 03:16:13.094622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.094584    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"]
Oct 30 03:16:13.128043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.128004    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-podinfo\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.128242 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.128133    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.128242 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.128208    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-certs\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.128374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.128300    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-credential-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.128374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.128336    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-data\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.128483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.128393    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98b7q\" (UniqueName: \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-kube-api-access-98b7q\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.128596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.128568    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-envoy\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.128682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.128666    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istiod-ca-cert\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.128761 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.128748    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-token\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230157 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230117    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-token\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230242    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-podinfo\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230300    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230322    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-certs\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230400    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-credential-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230450    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-data\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230477    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98b7q\" (UniqueName: \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-kube-api-access-98b7q\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230582    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-envoy\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230643    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istiod-ca-cert\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230868    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-credential-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.230973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.230908    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-data\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.231239 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.231202    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-certs\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.231356 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.231326    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.231618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.231557    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istiod-ca-cert\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.233334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.233306    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-envoy\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.233595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.233575    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-podinfo\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.250374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.250347    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-token\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.250531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.250518    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98b7q\" (UniqueName: \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-kube-api-access-98b7q\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.374508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.374391    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:13.702024 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.701396    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"]
Oct 30 03:16:13.717175 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:16:13.716782    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090d3f9c_73ac_4bf1_90e2_e0fc9e9db43b.slice/crio-c0fa19bcbb3e4d864156cd41b4ea849b22236e73b9c079ab266ccd9ae7af9395 WatchSource:0}: Error finding container c0fa19bcbb3e4d864156cd41b4ea849b22236e73b9c079ab266ccd9ae7af9395: Status 404 returned error can't find the container with id c0fa19bcbb3e4d864156cd41b4ea849b22236e73b9c079ab266ccd9ae7af9395
Oct 30 03:16:13.723816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.721870    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:16:13.723816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.721978    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:16:13.723816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:13.722040    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:16:14.040277 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:14.040224    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"]
Oct 30 03:16:14.044245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:14.044209    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990b694-tffjs"]
Oct 30 03:16:14.201797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:14.201762    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25bad01a-cef3-4d53-be60-2e2e0de1fc8e" path="/var/lib/kubelet/pods/25bad01a-cef3-4d53-be60-2e2e0de1fc8e/volumes"
Oct 30 03:16:14.511821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:14.511781    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" event={"ID":"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b","Type":"ContainerStarted","Data":"3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34"}
Oct 30 03:16:14.512015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:14.511841    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" event={"ID":"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b","Type":"ContainerStarted","Data":"c0fa19bcbb3e4d864156cd41b4ea849b22236e73b9c079ab266ccd9ae7af9395"}
Oct 30 03:16:14.557575 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:14.557522    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" podStartSLOduration=1.5575053909999999 podStartE2EDuration="1.557505391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:14.553948543 +0000 UTC m=+2112.478156731" watchObservedRunningTime="2026-10-30 03:16:14.557505391 +0000 UTC m=+2112.481713578"
Oct 30 03:16:15.014374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.014334    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"]
Oct 30 03:16:15.018590 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.018568    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.039241 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.039207    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"]
Oct 30 03:16:15.051285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.046790    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:16:15.051285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.046790    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:16:15.083707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.083656    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f75cbf9-ff6rp\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.083914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.083787    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f75cbf9-ff6rp\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.083967 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.083928    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6sbw\" (UniqueName: \"kubernetes.io/projected/378fd8b1-2e02-4e04-9688-6192a71ccf21-kube-api-access-v6sbw\") pod \"maas-api-revoke-sub-7f75cbf9-ff6rp\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.185951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.185897    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6sbw\" (UniqueName: \"kubernetes.io/projected/378fd8b1-2e02-4e04-9688-6192a71ccf21-kube-api-access-v6sbw\") pod \"maas-api-revoke-sub-7f75cbf9-ff6rp\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.186185 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.186166    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f75cbf9-ff6rp\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.186253 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.186240    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f75cbf9-ff6rp\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.187084 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.187063    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f75cbf9-ff6rp\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.189181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.189152    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f75cbf9-ff6rp\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.206568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.206535    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6sbw\" (UniqueName: \"kubernetes.io/projected/378fd8b1-2e02-4e04-9688-6192a71ccf21-kube-api-access-v6sbw\") pod \"maas-api-revoke-sub-7f75cbf9-ff6rp\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.334513 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.334395    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:15.375875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.375825    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:15.389564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.380511    4278 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:16:15.389772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.389598    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 03:16:15.693667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:15.693579    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"]
Oct 30 03:16:15.698703 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:16:15.698671    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378fd8b1_2e02_4e04_9688_6192a71ccf21.slice/crio-0b7be3041c019a92557cc9d4e8cc4cddf571e811c2d484ccde56e4fbb5d706b9 WatchSource:0}: Error finding container 0b7be3041c019a92557cc9d4e8cc4cddf571e811c2d484ccde56e4fbb5d706b9: Status 404 returned error can't find the container with id 0b7be3041c019a92557cc9d4e8cc4cddf571e811c2d484ccde56e4fbb5d706b9
Oct 30 03:16:16.384670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:16.384553    4278 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:16:16.384670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:16.384610    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 03:16:16.591155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:16.591109    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp" event={"ID":"378fd8b1-2e02-4e04-9688-6192a71ccf21","Type":"ContainerStarted","Data":"809352ad2e992ded028a64052780cfb23b6a1b9e45cfd93d458973ffce5b7cc1"}
Oct 30 03:16:16.591417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:16.591398    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp" event={"ID":"378fd8b1-2e02-4e04-9688-6192a71ccf21","Type":"ContainerStarted","Data":"0b7be3041c019a92557cc9d4e8cc4cddf571e811c2d484ccde56e4fbb5d706b9"}
Oct 30 03:16:16.631469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:16.631366    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp" podStartSLOduration=2.631349331 podStartE2EDuration="2.631349331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:16.629611619 +0000 UTC m=+2114.553819818" watchObservedRunningTime="2026-10-30 03:16:16.631349331 +0000 UTC m=+2114.555557519"
Oct 30 03:16:17.387454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:17.378758    4278 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:16:17.387454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:17.378835    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 03:16:17.601046 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:17.601002    4278 generic.go:356] "Generic (PLEG): container finished" podID="378fd8b1-2e02-4e04-9688-6192a71ccf21" containerID="809352ad2e992ded028a64052780cfb23b6a1b9e45cfd93d458973ffce5b7cc1" exitCode=0
Oct 30 03:16:17.601245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:17.601084    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp" event={"ID":"378fd8b1-2e02-4e04-9688-6192a71ccf21","Type":"ContainerDied","Data":"809352ad2e992ded028a64052780cfb23b6a1b9e45cfd93d458973ffce5b7cc1"}
Oct 30 03:16:18.377867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:18.377822    4278 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:16:18.378173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:18.378141    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 03:16:19.090935 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.090895    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:19.193974 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.193926    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-openshift-ca\") pod \"378fd8b1-2e02-4e04-9688-6192a71ccf21\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") "
Oct 30 03:16:19.194364 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.194341    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378fd8b1-2e02-4e04-9688-6192a71ccf21-kube-api-access-v6sbw\" (UniqueName: \"kubernetes.io/projected/378fd8b1-2e02-4e04-9688-6192a71ccf21-kube-api-access-v6sbw\") pod \"378fd8b1-2e02-4e04-9688-6192a71ccf21\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") "
Oct 30 03:16:19.194549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.194519    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-xks-ca\") pod \"378fd8b1-2e02-4e04-9688-6192a71ccf21\" (UID: \"378fd8b1-2e02-4e04-9688-6192a71ccf21\") "
Oct 30 03:16:19.195119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.195081    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-openshift-ca" pod "378fd8b1-2e02-4e04-9688-6192a71ccf21" (UID: "378fd8b1-2e02-4e04-9688-6192a71ccf21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:19.196398 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.196375    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:19.198297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.198189    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-xks-ca" pod "378fd8b1-2e02-4e04-9688-6192a71ccf21" (UID: "378fd8b1-2e02-4e04-9688-6192a71ccf21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:19.198297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.198217    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378fd8b1-2e02-4e04-9688-6192a71ccf21-kube-api-access-v6sbw" pod "378fd8b1-2e02-4e04-9688-6192a71ccf21" (UID: "378fd8b1-2e02-4e04-9688-6192a71ccf21"). InnerVolumeSpecName "kube-api-access-v6sbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:19.298466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.298369    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6sbw\" (UniqueName: \"kubernetes.io/projected/378fd8b1-2e02-4e04-9688-6192a71ccf21-kube-api-access-v6sbw\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:19.298466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.298399    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378fd8b1-2e02-4e04-9688-6192a71ccf21-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:19.383446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.381135    4278 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:16:19.383446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.381198    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 03:16:19.640067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.632584    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp" event={"ID":"378fd8b1-2e02-4e04-9688-6192a71ccf21","Type":"ContainerDied","Data":"0b7be3041c019a92557cc9d4e8cc4cddf571e811c2d484ccde56e4fbb5d706b9"}
Oct 30 03:16:19.640067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.632658    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b7be3041c019a92557cc9d4e8cc4cddf571e811c2d484ccde56e4fbb5d706b9"
Oct 30 03:16:19.640067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:19.632738    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"
Oct 30 03:16:20.375371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.375329    4278 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:16:20.375939 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.375414    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 03:16:20.459887 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.459700    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"]
Oct 30 03:16:20.462708 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.462662    4278 cpu_manager.go:414] "containerMap: removing container" podUID="378fd8b1-2e02-4e04-9688-6192a71ccf21" containerName="revoke-keys"
Oct 30 03:16:20.462896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.462742    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378fd8b1-2e02-4e04-9688-6192a71ccf21" containerName="revoke-keys"
Oct 30 03:16:20.463007 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.462983    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="378fd8b1-2e02-4e04-9688-6192a71ccf21" containerName="revoke-keys"
Oct 30 03:16:20.467495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.467468    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.483876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.480889    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"]
Oct 30 03:16:20.495445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.495275    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4cf8f5-serving-cert\""
Oct 30 03:16:20.556695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.556659    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdgt\" (UniqueName: \"kubernetes.io/projected/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-kube-api-access-lkdgt\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.556894 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.556869    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.556961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.556922    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-metrics-certs\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.557047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.557027    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-maas-api-tls\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.658803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.658399    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.658803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.658522    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-metrics-certs\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.658803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.658652    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-maas-api-tls\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.659197 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.659060    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdgt\" (UniqueName: \"kubernetes.io/projected/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-kube-api-access-lkdgt\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.659360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.659337    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.662118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.662090    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-metrics-certs\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.662670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.662644    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-maas-api-tls\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.680416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.680381    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdgt\" (UniqueName: \"kubernetes.io/projected/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-kube-api-access-lkdgt\") pod \"maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:20.796458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:20.790760    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:21.137605 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:16:21.137567    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e12b7f9_c9e6_4aea_9b62_4031f0c71af3.slice/crio-abda92064dae1848c559e759056d476d19197de8b1c4bcfc55c1722b2e11c555 WatchSource:0}: Error finding container abda92064dae1848c559e759056d476d19197de8b1c4bcfc55c1722b2e11c555: Status 404 returned error can't find the container with id abda92064dae1848c559e759056d476d19197de8b1c4bcfc55c1722b2e11c555
Oct 30 03:16:21.138799 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:21.138763    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"]
Oct 30 03:16:21.139298 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:21.139280    4278 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:16:21.456404 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:21.456303    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:21.457188 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:21.457100    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:21.458557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:21.458535    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:16:21.655807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:21.655772    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6" event={"ID":"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3","Type":"ContainerStarted","Data":"abda92064dae1848c559e759056d476d19197de8b1c4bcfc55c1722b2e11c555"}
Oct 30 03:16:22.665659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:22.665607    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6" event={"ID":"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3","Type":"ContainerStarted","Data":"26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361"}
Oct 30 03:16:22.666198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:22.665954    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:22.708654 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:22.708592    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6" podStartSLOduration=2.221807347 podStartE2EDuration="2.708577985s" totalImagesPullingTime="486.770638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:22.705768267 +0000 UTC m=+2120.629976455" watchObservedRunningTime="2026-10-30 03:16:22.708577985 +0000 UTC m=+2120.632786173"
Oct 30 03:16:23.699292 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.699254    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"]
Oct 30 03:16:23.704950 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.704923    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.724214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.724186    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"]
Oct 30 03:16:23.730691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.730665    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ac7936\"/\"kube-root-ca.crt\""
Oct 30 03:16:23.730886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.730698    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ac7936\"/\"sub-test-model-ac7936-kserve-self-signed-certs\""
Oct 30 03:16:23.730971 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.730792    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ac7936\"/\"openshift-service-ca.crt\""
Oct 30 03:16:23.735395 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.735373    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ac7936\"/\"default-dockercfg-q26wl\""
Oct 30 03:16:23.821378 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.821051    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f9f398-a946-403d-89ff-82a803d6f639-tls-certs\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.821378 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.821225    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-home\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.821378 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.821335    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-tmp-dir\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.821708 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.821538    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84vj9\" (UniqueName: \"kubernetes.io/projected/b4f9f398-a946-403d-89ff-82a803d6f639-kube-api-access-84vj9\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.821708 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.821578    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-dshm\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.821853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.821833    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-model-cache\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.923645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.923601    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-home\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.923830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.923710    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-tmp-dir\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.923921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.923838    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84vj9\" (UniqueName: \"kubernetes.io/projected/b4f9f398-a946-403d-89ff-82a803d6f639-kube-api-access-84vj9\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.923921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.923869    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-dshm\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.924082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.924060    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-model-cache\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.924174 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.924107    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-home\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.924174 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.924123    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f9f398-a946-403d-89ff-82a803d6f639-tls-certs\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.924287 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.924186    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-tmp-dir\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.924390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.924362    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-model-cache\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.926623 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.926596    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-dshm\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.926978 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.926953    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f9f398-a946-403d-89ff-82a803d6f639-tls-certs\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:23.982524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:23.980230    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84vj9\" (UniqueName: \"kubernetes.io/projected/b4f9f398-a946-403d-89ff-82a803d6f639-kube-api-access-84vj9\") pod \"sub-test-model-ac7936-kserve-5787685d7f-nsmdm\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:24.044454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:24.035219    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:24.223557 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:16:24.223503    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:16:24.440398 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:16:24.440363    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4f9f398_a946_403d_89ff_82a803d6f639.slice/crio-4f900e9dd74387a88de0edaf0d0903b041e0f773fbad16e9b5849c4e51acaf06 WatchSource:0}: Error finding container 4f900e9dd74387a88de0edaf0d0903b041e0f773fbad16e9b5849c4e51acaf06: Status 404 returned error can't find the container with id 4f900e9dd74387a88de0edaf0d0903b041e0f773fbad16e9b5849c4e51acaf06
Oct 30 03:16:24.441240 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:24.441206    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"]
Oct 30 03:16:24.685874 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:24.685834    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm" event={"ID":"b4f9f398-a946-403d-89ff-82a803d6f639","Type":"ContainerStarted","Data":"2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3"}
Oct 30 03:16:24.686085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:24.685884    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm" event={"ID":"b4f9f398-a946-403d-89ff-82a803d6f639","Type":"ContainerStarted","Data":"4f900e9dd74387a88de0edaf0d0903b041e0f773fbad16e9b5849c4e51acaf06"}
Oct 30 03:16:24.729086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:24.728975    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm" podStartSLOduration=1.728959304 podStartE2EDuration="1.728959304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:24.725665702 +0000 UTC m=+2122.649873883" watchObservedRunningTime="2026-10-30 03:16:24.728959304 +0000 UTC m=+2122.653167492"
Oct 30 03:16:25.037725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:25.037684    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:28.682574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:28.682538    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:16:32.613707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.613670    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"]
Oct 30 03:16:32.617832 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.617809    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:32.636528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.636499    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"]
Oct 30 03:16:32.644100 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.644067    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:16:32.644484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.644463    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:16:32.688214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.688173    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33af9be9-tcddw\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:32.688407 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.688321    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33af9be9-tcddw\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:32.688550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.688519    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7llf\" (UniqueName: \"kubernetes.io/projected/c0509c78-4b5d-4062-814b-3a37462527ac-kube-api-access-j7llf\") pod \"maas-api-revoke-sub-33af9be9-tcddw\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:32.790202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.790160    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33af9be9-tcddw\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:32.790527 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.790264    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7llf\" (UniqueName: \"kubernetes.io/projected/c0509c78-4b5d-4062-814b-3a37462527ac-kube-api-access-j7llf\") pod \"maas-api-revoke-sub-33af9be9-tcddw\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:32.790527 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.790458    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33af9be9-tcddw\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:32.791108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.791087    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33af9be9-tcddw\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:32.792825 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.792803    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33af9be9-tcddw\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:32.810924 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.810896    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7llf\" (UniqueName: \"kubernetes.io/projected/c0509c78-4b5d-4062-814b-3a37462527ac-kube-api-access-j7llf\") pod \"maas-api-revoke-sub-33af9be9-tcddw\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:32.935020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:32.934921    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:33.240083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:33.240042    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"]
Oct 30 03:16:33.241186 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:16:33.241151    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0509c78_4b5d_4062_814b_3a37462527ac.slice/crio-7db8542c4adfd06c875a84ff8a1f60e909557b13fa3194ea4a4ad5d6a110258c WatchSource:0}: Error finding container 7db8542c4adfd06c875a84ff8a1f60e909557b13fa3194ea4a4ad5d6a110258c: Status 404 returned error can't find the container with id 7db8542c4adfd06c875a84ff8a1f60e909557b13fa3194ea4a4ad5d6a110258c
Oct 30 03:16:33.758495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:33.758458    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw" event={"ID":"c0509c78-4b5d-4062-814b-3a37462527ac","Type":"ContainerStarted","Data":"7db8542c4adfd06c875a84ff8a1f60e909557b13fa3194ea4a4ad5d6a110258c"}
Oct 30 03:16:34.036061 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:34.036026    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:34.049800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:34.049769    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:34.768622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:34.768586    4278 generic.go:356] "Generic (PLEG): container finished" podID="c0509c78-4b5d-4062-814b-3a37462527ac" containerID="5224c8962f7e448d38e1310118a0230621171e8b4c54a53e66b5b688aaecb195" exitCode=0
Oct 30 03:16:34.769158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:34.768663    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw" event={"ID":"c0509c78-4b5d-4062-814b-3a37462527ac","Type":"ContainerDied","Data":"5224c8962f7e448d38e1310118a0230621171e8b4c54a53e66b5b688aaecb195"}
Oct 30 03:16:34.780784 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:34.780743    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:16:35.954283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:35.954257    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:36.046075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.045974    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-xks-ca\") pod \"c0509c78-4b5d-4062-814b-3a37462527ac\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") "
Oct 30 03:16:36.046075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.046034    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0509c78-4b5d-4062-814b-3a37462527ac-kube-api-access-j7llf\" (UniqueName: \"kubernetes.io/projected/c0509c78-4b5d-4062-814b-3a37462527ac-kube-api-access-j7llf\") pod \"c0509c78-4b5d-4062-814b-3a37462527ac\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") "
Oct 30 03:16:36.046325 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.046092    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-openshift-ca\") pod \"c0509c78-4b5d-4062-814b-3a37462527ac\" (UID: \"c0509c78-4b5d-4062-814b-3a37462527ac\") "
Oct 30 03:16:36.046562 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.046533    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-openshift-ca" pod "c0509c78-4b5d-4062-814b-3a37462527ac" (UID: "c0509c78-4b5d-4062-814b-3a37462527ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:36.046863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.046843    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:36.048274 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.048239    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-xks-ca" pod "c0509c78-4b5d-4062-814b-3a37462527ac" (UID: "c0509c78-4b5d-4062-814b-3a37462527ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:36.048274 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.048255    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0509c78-4b5d-4062-814b-3a37462527ac-kube-api-access-j7llf" pod "c0509c78-4b5d-4062-814b-3a37462527ac" (UID: "c0509c78-4b5d-4062-814b-3a37462527ac"). InnerVolumeSpecName "kube-api-access-j7llf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:36.150103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.150066    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0509c78-4b5d-4062-814b-3a37462527ac-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:36.150348 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.150332    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7llf\" (UniqueName: \"kubernetes.io/projected/c0509c78-4b5d-4062-814b-3a37462527ac-kube-api-access-j7llf\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:16:36.786106 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.786071    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"
Oct 30 03:16:36.786275 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.786104    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw" event={"ID":"c0509c78-4b5d-4062-814b-3a37462527ac","Type":"ContainerDied","Data":"7db8542c4adfd06c875a84ff8a1f60e909557b13fa3194ea4a4ad5d6a110258c"}
Oct 30 03:16:36.786275 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:36.786153    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7db8542c4adfd06c875a84ff8a1f60e909557b13fa3194ea4a4ad5d6a110258c"
Oct 30 03:16:49.557528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:16:49.557484    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:16:54.197286 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:16:54.197238    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:17:03.949673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:03.949636    4278 scope.go:122] "RemoveContainer" containerID="54832ee542fa59bfdef13dbd60bfc936f7f55db7e1820e082cfbbc85190533bb"
Oct 30 03:17:03.961052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:03.961024    4278 scope.go:122] "RemoveContainer" containerID="c238435e4ee2a1de1e5784aa3d41ac12ece75cd12c1b569ad3e5844dc0bc93cb"
Oct 30 03:17:04.351358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:04.351312    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"]
Oct 30 03:17:05.083512 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.083449    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"]
Oct 30 03:17:05.084151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.084136    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c0509c78-4b5d-4062-814b-3a37462527ac" containerName="revoke-keys"
Oct 30 03:17:05.084206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.084153    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0509c78-4b5d-4062-814b-3a37462527ac" containerName="revoke-keys"
Oct 30 03:17:05.084319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.084309    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0509c78-4b5d-4062-814b-3a37462527ac" containerName="revoke-keys"
Oct 30 03:17:05.089910 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.089885    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.106471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.106440    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"]
Oct 30 03:17:05.114392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.114356    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:17:05.114585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.114461    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:17:05.154532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.154492    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.154728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.154694    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.154872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.154850    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87bg\" (UniqueName: \"kubernetes.io/projected/f72e37ed-cf22-44b0-a5f2-314b860d7f78-kube-api-access-x87bg\") pod \"maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.270060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.260682    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.271367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.271327    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.272555 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.262938    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.272995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.272968    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87bg\" (UniqueName: \"kubernetes.io/projected/f72e37ed-cf22-44b0-a5f2-314b860d7f78-kube-api-access-x87bg\") pod \"maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.276515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.276490    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.293873 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.293841    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87bg\" (UniqueName: \"kubernetes.io/projected/f72e37ed-cf22-44b0-a5f2-314b860d7f78-kube-api-access-x87bg\") pod \"maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.405593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.405505    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:05.462930 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.462892    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"]
Oct 30 03:17:05.470045 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.470022    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:05.502933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.502901    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"]
Oct 30 03:17:05.578086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.578052    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8acd081b-9gjzs\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:05.578283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.578183    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c97td\" (UniqueName: \"kubernetes.io/projected/9b621ddc-d52c-41e6-a9be-63d8008f40bb-kube-api-access-c97td\") pod \"maas-api-revoke-sub-8acd081b-9gjzs\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:05.578419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.578395    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8acd081b-9gjzs\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:05.680139 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.680037    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8acd081b-9gjzs\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:05.680300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.680188    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8acd081b-9gjzs\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:05.680300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.680282    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c97td\" (UniqueName: \"kubernetes.io/projected/9b621ddc-d52c-41e6-a9be-63d8008f40bb-kube-api-access-c97td\") pod \"maas-api-revoke-sub-8acd081b-9gjzs\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:05.680981 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.680955    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8acd081b-9gjzs\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:05.682604 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.682586    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8acd081b-9gjzs\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:05.699976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.699939    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c97td\" (UniqueName: \"kubernetes.io/projected/9b621ddc-d52c-41e6-a9be-63d8008f40bb-kube-api-access-c97td\") pod \"maas-api-revoke-sub-8acd081b-9gjzs\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:05.719206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.719160    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"]
Oct 30 03:17:05.721298 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:17:05.721267    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72e37ed_cf22_44b0_a5f2_314b860d7f78.slice/crio-737a50a116878b1f931e93a44fc946dd8a4c54af72c824801d07c9c6787d6658 WatchSource:0}: Error finding container 737a50a116878b1f931e93a44fc946dd8a4c54af72c824801d07c9c6787d6658: Status 404 returned error can't find the container with id 737a50a116878b1f931e93a44fc946dd8a4c54af72c824801d07c9c6787d6658
Oct 30 03:17:05.812469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:05.812415    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:06.066966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:06.066925    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt" event={"ID":"f72e37ed-cf22-44b0-a5f2-314b860d7f78","Type":"ContainerStarted","Data":"2efed3400f59b8e12e231dcc4df1f11146131fdd0b842b57091b0b394a6552a9"}
Oct 30 03:17:06.067150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:06.067018    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt" event={"ID":"f72e37ed-cf22-44b0-a5f2-314b860d7f78","Type":"ContainerStarted","Data":"737a50a116878b1f931e93a44fc946dd8a4c54af72c824801d07c9c6787d6658"}
Oct 30 03:17:06.109449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:06.109376    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt" podStartSLOduration=1.109357968 podStartE2EDuration="1.109357968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:17:06.106771227 +0000 UTC m=+2164.030979415" watchObservedRunningTime="2026-10-30 03:17:06.109357968 +0000 UTC m=+2164.033566157"
Oct 30 03:17:06.124611 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:06.124582    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"]
Oct 30 03:17:06.125229 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:17:06.125205    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b621ddc_d52c_41e6_a9be_63d8008f40bb.slice/crio-b6dc5ff623ecd389be06ffc54c8291d0c80f6f439095670a781f4aa813c66cc2 WatchSource:0}: Error finding container b6dc5ff623ecd389be06ffc54c8291d0c80f6f439095670a781f4aa813c66cc2: Status 404 returned error can't find the container with id b6dc5ff623ecd389be06ffc54c8291d0c80f6f439095670a781f4aa813c66cc2
Oct 30 03:17:07.076733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:07.076635    4278 generic.go:356] "Generic (PLEG): container finished" podID="f72e37ed-cf22-44b0-a5f2-314b860d7f78" containerID="2efed3400f59b8e12e231dcc4df1f11146131fdd0b842b57091b0b394a6552a9" exitCode=0
Oct 30 03:17:07.076918 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:07.076722    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt" event={"ID":"f72e37ed-cf22-44b0-a5f2-314b860d7f78","Type":"ContainerDied","Data":"2efed3400f59b8e12e231dcc4df1f11146131fdd0b842b57091b0b394a6552a9"}
Oct 30 03:17:07.078495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:07.078468    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs" event={"ID":"9b621ddc-d52c-41e6-a9be-63d8008f40bb","Type":"ContainerStarted","Data":"e282d00b3699da571396e09df90e22e096d2c4cf162f037676159862d0b5635b"}
Oct 30 03:17:07.078634 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:07.078513    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs" event={"ID":"9b621ddc-d52c-41e6-a9be-63d8008f40bb","Type":"ContainerStarted","Data":"b6dc5ff623ecd389be06ffc54c8291d0c80f6f439095670a781f4aa813c66cc2"}
Oct 30 03:17:07.160378 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:07.160315    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs" podStartSLOduration=2.160298202 podStartE2EDuration="2.160298202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:17:07.158248431 +0000 UTC m=+2165.082456619" watchObservedRunningTime="2026-10-30 03:17:07.160298202 +0000 UTC m=+2165.084506390"
Oct 30 03:17:08.088469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.088384    4278 generic.go:356] "Generic (PLEG): container finished" podID="9b621ddc-d52c-41e6-a9be-63d8008f40bb" containerID="e282d00b3699da571396e09df90e22e096d2c4cf162f037676159862d0b5635b" exitCode=0
Oct 30 03:17:08.088668 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.088471    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs" event={"ID":"9b621ddc-d52c-41e6-a9be-63d8008f40bb","Type":"ContainerDied","Data":"e282d00b3699da571396e09df90e22e096d2c4cf162f037676159862d0b5635b"}
Oct 30 03:17:08.285336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.285301    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:08.329800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.328304    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-openshift-ca\") pod \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") "
Oct 30 03:17:08.331040 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.330990    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-openshift-ca" pod "f72e37ed-cf22-44b0-a5f2-314b860d7f78" (UID: "f72e37ed-cf22-44b0-a5f2-314b860d7f78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:08.337256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.328569    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-xks-ca\") pod \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") "
Oct 30 03:17:08.337869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.337838    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72e37ed-cf22-44b0-a5f2-314b860d7f78-kube-api-access-x87bg\" (UniqueName: \"kubernetes.io/projected/f72e37ed-cf22-44b0-a5f2-314b860d7f78-kube-api-access-x87bg\") pod \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\" (UID: \"f72e37ed-cf22-44b0-a5f2-314b860d7f78\") "
Oct 30 03:17:08.339054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.338989    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:08.352803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.352755    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-xks-ca" pod "f72e37ed-cf22-44b0-a5f2-314b860d7f78" (UID: "f72e37ed-cf22-44b0-a5f2-314b860d7f78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:17:08.353050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.352851    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72e37ed-cf22-44b0-a5f2-314b860d7f78-kube-api-access-x87bg" pod "f72e37ed-cf22-44b0-a5f2-314b860d7f78" (UID: "f72e37ed-cf22-44b0-a5f2-314b860d7f78"). InnerVolumeSpecName "kube-api-access-x87bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:08.441254 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.441207    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72e37ed-cf22-44b0-a5f2-314b860d7f78-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:08.441254 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:08.441249    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x87bg\" (UniqueName: \"kubernetes.io/projected/f72e37ed-cf22-44b0-a5f2-314b860d7f78-kube-api-access-x87bg\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:09.106850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.106178    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"
Oct 30 03:17:09.106850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.106195    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt" event={"ID":"f72e37ed-cf22-44b0-a5f2-314b860d7f78","Type":"ContainerDied","Data":"737a50a116878b1f931e93a44fc946dd8a4c54af72c824801d07c9c6787d6658"}
Oct 30 03:17:09.106850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.106261    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="737a50a116878b1f931e93a44fc946dd8a4c54af72c824801d07c9c6787d6658"
Oct 30 03:17:09.306362 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.306332    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:09.357915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.357832    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-openshift-ca\") pod \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") "
Oct 30 03:17:09.357915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.357891    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-xks-ca\") pod \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") "
Oct 30 03:17:09.358108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.358010    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b621ddc-d52c-41e6-a9be-63d8008f40bb-kube-api-access-c97td\" (UniqueName: \"kubernetes.io/projected/9b621ddc-d52c-41e6-a9be-63d8008f40bb-kube-api-access-c97td\") pod \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\" (UID: \"9b621ddc-d52c-41e6-a9be-63d8008f40bb\") "
Oct 30 03:17:09.369904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.358912    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-openshift-ca" pod "9b621ddc-d52c-41e6-a9be-63d8008f40bb" (UID: "9b621ddc-d52c-41e6-a9be-63d8008f40bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:09.369904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.359869    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:09.369904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.364809    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b621ddc-d52c-41e6-a9be-63d8008f40bb-kube-api-access-c97td" pod "9b621ddc-d52c-41e6-a9be-63d8008f40bb" (UID: "9b621ddc-d52c-41e6-a9be-63d8008f40bb"). InnerVolumeSpecName "kube-api-access-c97td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:09.376406 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.376374    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-xks-ca" pod "9b621ddc-d52c-41e6-a9be-63d8008f40bb" (UID: "9b621ddc-d52c-41e6-a9be-63d8008f40bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:17:09.462797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.462720    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b621ddc-d52c-41e6-a9be-63d8008f40bb-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:09.462797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:09.462770    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c97td\" (UniqueName: \"kubernetes.io/projected/9b621ddc-d52c-41e6-a9be-63d8008f40bb-kube-api-access-c97td\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:10.115633 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:10.115597    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"
Oct 30 03:17:10.115633 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:10.115606    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs" event={"ID":"9b621ddc-d52c-41e6-a9be-63d8008f40bb","Type":"ContainerDied","Data":"b6dc5ff623ecd389be06ffc54c8291d0c80f6f439095670a781f4aa813c66cc2"}
Oct 30 03:17:10.115868 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:10.115695    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6dc5ff623ecd389be06ffc54c8291d0c80f6f439095670a781f4aa813c66cc2"
Oct 30 03:17:16.066921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.066878    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"]
Oct 30 03:17:16.067775 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.067750    4278 cpu_manager.go:414] "containerMap: removing container" podUID="9b621ddc-d52c-41e6-a9be-63d8008f40bb" containerName="revoke-keys"
Oct 30 03:17:16.067775 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.067772    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b621ddc-d52c-41e6-a9be-63d8008f40bb" containerName="revoke-keys"
Oct 30 03:17:16.067947 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.067833    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f72e37ed-cf22-44b0-a5f2-314b860d7f78" containerName="revoke-keys"
Oct 30 03:17:16.067947 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.067839    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72e37ed-cf22-44b0-a5f2-314b860d7f78" containerName="revoke-keys"
Oct 30 03:17:16.068016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.067990    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72e37ed-cf22-44b0-a5f2-314b860d7f78" containerName="revoke-keys"
Oct 30 03:17:16.068016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.068004    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b621ddc-d52c-41e6-a9be-63d8008f40bb" containerName="revoke-keys"
Oct 30 03:17:16.071960 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.071940    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.102056 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.102009    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c1fce3-openshift-default-dockercfg-l84qc\""
Oct 30 03:17:16.106518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.106485    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"]
Oct 30 03:17:16.182481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.170966    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.182481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.171042    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-credential-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.182481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.171086    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-envoy\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.182481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.171128    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-certs\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.182481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.171213    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-token\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.182481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.171310    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxzq\" (UniqueName: \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-kube-api-access-7sxzq\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.182481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.171395    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-data\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.182481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.171575    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcb3dd9-89f6-4a93-997f-dec09895c095-istiod-ca-cert\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.182481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.171613    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-podinfo\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.274533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.274486    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxzq\" (UniqueName: \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-kube-api-access-7sxzq\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.275210 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.275181    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-data\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.275543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.275524    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcb3dd9-89f6-4a93-997f-dec09895c095-istiod-ca-cert\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.275695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.275682    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-podinfo\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.275901 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.275888    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.276007 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.275996    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-credential-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.276103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.276089    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-envoy\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.276200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.276188    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-certs\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.276338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.276324    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-token\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.277177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.277153    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-data\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.277933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.277914    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcb3dd9-89f6-4a93-997f-dec09895c095-istiod-ca-cert\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.287504 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.278766    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-credential-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.287504 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.279013    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.288102 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.288051    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-certs\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.301446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.292746    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-envoy\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.310355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.310273    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxzq\" (UniqueName: \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-kube-api-access-7sxzq\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.319497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.313942    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-podinfo\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.348460 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.327247    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-token\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.387073 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.387036    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:16.710567 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.710537    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"]
Oct 30 03:17:16.712195 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:17:16.712155    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcb3dd9_89f6_4a93_997f_dec09895c095.slice/crio-8a078dc17a802d011e0f68c3a36bf4757243e304a4c8d20668f1ebf3030735f1 WatchSource:0}: Error finding container 8a078dc17a802d011e0f68c3a36bf4757243e304a4c8d20668f1ebf3030735f1: Status 404 returned error can't find the container with id 8a078dc17a802d011e0f68c3a36bf4757243e304a4c8d20668f1ebf3030735f1
Oct 30 03:17:16.715255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.715217    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:17:16.715365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.715294    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:17:16.715365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:16.715325    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:17:17.199658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:17.199613    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" event={"ID":"adcb3dd9-89f6-4a93-997f-dec09895c095","Type":"ContainerStarted","Data":"0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c"}
Oct 30 03:17:17.200298 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:17.200217    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" event={"ID":"adcb3dd9-89f6-4a93-997f-dec09895c095","Type":"ContainerStarted","Data":"8a078dc17a802d011e0f68c3a36bf4757243e304a4c8d20668f1ebf3030735f1"}
Oct 30 03:17:17.247009 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:17.246948    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" podStartSLOduration=1.246931089 podStartE2EDuration="1.246931089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:17:17.243350684 +0000 UTC m=+2175.167558872" watchObservedRunningTime="2026-10-30 03:17:17.246931089 +0000 UTC m=+2175.171139345"
Oct 30 03:17:17.388023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:17.387981    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:17.390500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:17.390466    4278 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:17:17.390658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:17.390527    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 03:17:18.388070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:18.388030    4278 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:17:18.388534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:18.388108    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 03:17:18.600273 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:18.600235    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"]
Oct 30 03:17:18.600491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:18.600455    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy" containerID="cri-o://6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712" gracePeriod=30
Oct 30 03:17:19.383455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.382597    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f" podUID="2a39628b-20dd-4169-a2f5-672963350cfa" containerName="main" containerID="cri-o://4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73" gracePeriod=45
Oct 30 03:17:19.387660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.387624    4278 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:17:19.387814 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.387687    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 03:17:19.732000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.731968    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:17:19.836163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.836122    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-model-cache\") pod \"2a39628b-20dd-4169-a2f5-672963350cfa\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") "
Oct 30 03:17:19.836696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.836280    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a39628b-20dd-4169-a2f5-672963350cfa-kube-api-access-bmrs9\" (UniqueName: \"kubernetes.io/projected/2a39628b-20dd-4169-a2f5-672963350cfa-kube-api-access-bmrs9\") pod \"2a39628b-20dd-4169-a2f5-672963350cfa\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") "
Oct 30 03:17:19.836696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.836398    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-home\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-home\") pod \"2a39628b-20dd-4169-a2f5-672963350cfa\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") "
Oct 30 03:17:19.836696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.836472    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-model-cache" pod "2a39628b-20dd-4169-a2f5-672963350cfa" (UID: "2a39628b-20dd-4169-a2f5-672963350cfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:19.836696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.836489    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-tmp-dir\") pod \"2a39628b-20dd-4169-a2f5-672963350cfa\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") "
Oct 30 03:17:19.836696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.836529    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a39628b-20dd-4169-a2f5-672963350cfa-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a39628b-20dd-4169-a2f5-672963350cfa-tls-certs\") pod \"2a39628b-20dd-4169-a2f5-672963350cfa\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") "
Oct 30 03:17:19.836696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.836639    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-dshm\") pod \"2a39628b-20dd-4169-a2f5-672963350cfa\" (UID: \"2a39628b-20dd-4169-a2f5-672963350cfa\") "
Oct 30 03:17:19.837896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.837007    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-tmp-dir" pod "2a39628b-20dd-4169-a2f5-672963350cfa" (UID: "2a39628b-20dd-4169-a2f5-672963350cfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:19.837896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.837481    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-home" pod "2a39628b-20dd-4169-a2f5-672963350cfa" (UID: "2a39628b-20dd-4169-a2f5-672963350cfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:19.837896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.837645    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:19.837896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.837666    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:19.837896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.837681    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:19.840384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.840355    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a39628b-20dd-4169-a2f5-672963350cfa-tls-certs" pod "2a39628b-20dd-4169-a2f5-672963350cfa" (UID: "2a39628b-20dd-4169-a2f5-672963350cfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:17:19.840600 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.840577    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-dshm" pod "2a39628b-20dd-4169-a2f5-672963350cfa" (UID: "2a39628b-20dd-4169-a2f5-672963350cfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:19.841771 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.841748    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a39628b-20dd-4169-a2f5-672963350cfa-kube-api-access-bmrs9" pod "2a39628b-20dd-4169-a2f5-672963350cfa" (UID: "2a39628b-20dd-4169-a2f5-672963350cfa"). InnerVolumeSpecName "kube-api-access-bmrs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:19.939516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.939409    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a39628b-20dd-4169-a2f5-672963350cfa-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:19.939516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.939453    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a39628b-20dd-4169-a2f5-672963350cfa-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:19.939516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:19.939463    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmrs9\" (UniqueName: \"kubernetes.io/projected/2a39628b-20dd-4169-a2f5-672963350cfa-kube-api-access-bmrs9\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:20.222262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.222225    4278 generic.go:356] "Generic (PLEG): container finished" podID="2a39628b-20dd-4169-a2f5-672963350cfa" containerID="4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73" exitCode=0
Oct 30 03:17:20.222438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.222272    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"
Oct 30 03:17:20.222438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.222301    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f" event={"ID":"2a39628b-20dd-4169-a2f5-672963350cfa","Type":"ContainerDied","Data":"4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73"}
Oct 30 03:17:20.222438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.222346    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f" event={"ID":"2a39628b-20dd-4169-a2f5-672963350cfa","Type":"ContainerDied","Data":"dbbc4c07228471aa9f243f7f5013b3c1fe24383e8ed520411d7912ca961513e8"}
Oct 30 03:17:20.222438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.222363    4278 scope.go:122] "RemoveContainer" containerID="4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73"
Oct 30 03:17:20.239573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.239519    4278 scope.go:122] "RemoveContainer" containerID="4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73"
Oct 30 03:17:20.239881 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:17:20.239860    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73\": container with ID starting with 4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73 not found: ID does not exist" containerID="4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73"
Oct 30 03:17:20.239989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.239890    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73"} err="failed to get container status \"4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73\": rpc error: code = NotFound desc = could not find container \"4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73\": container with ID starting with 4dc1a742b51bcc64c45d9d6a9ac0964d8fbcfcc11e8bcd9cc7d0f68743db2a73 not found: ID does not exist"
Oct 30 03:17:20.291851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.291813    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"]
Oct 30 03:17:20.302223 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.302193    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f"]
Oct 30 03:17:20.389827 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.389788    4278 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:17:20.390010 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.389845    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 03:17:20.606837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:20.606541    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4f415d/test-model-4f415d-kserve-64fbb5585f-gqn6f" podUID="2a39628b-20dd-4169-a2f5-672963350cfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.157:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:17:21.387993 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:21.387918    4278 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:17:21.388489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:21.388010    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 03:17:22.214501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:22.214416    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a39628b-20dd-4169-a2f5-672963350cfa" path="/var/lib/kubelet/pods/2a39628b-20dd-4169-a2f5-672963350cfa/volumes"
Oct 30 03:17:22.387550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:22.387511    4278 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:17:22.387796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:22.387775    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 03:17:22.528913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:22.528872    4278 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:17:22.529308 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:22.528931    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:17:23.429119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:23.429084    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:23.429914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:23.429878    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:23.431478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:23.431454    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:23.912522 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:23.912491    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 03:17:24.029368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029324    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-certs\") pod \"443f59af-9b47-43ea-91b1-63054fcd0ec7\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") "
Oct 30 03:17:24.029563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029457    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-podinfo\") pod \"443f59af-9b47-43ea-91b1-63054fcd0ec7\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") "
Oct 30 03:17:24.029563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029549    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-data\") pod \"443f59af-9b47-43ea-91b1-63054fcd0ec7\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") "
Oct 30 03:17:24.029791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029670    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-token\" (UniqueName: \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-token\") pod \"443f59af-9b47-43ea-91b1-63054fcd0ec7\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") "
Oct 30 03:17:24.029791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029680    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-certs" pod "443f59af-9b47-43ea-91b1-63054fcd0ec7" (UID: "443f59af-9b47-43ea-91b1-63054fcd0ec7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:24.029791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029714    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-envoy\") pod \"443f59af-9b47-43ea-91b1-63054fcd0ec7\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") "
Oct 30 03:17:24.029791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029760    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/443f59af-9b47-43ea-91b1-63054fcd0ec7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443f59af-9b47-43ea-91b1-63054fcd0ec7-istiod-ca-cert\") pod \"443f59af-9b47-43ea-91b1-63054fcd0ec7\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") "
Oct 30 03:17:24.029997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029807    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-socket\") pod \"443f59af-9b47-43ea-91b1-63054fcd0ec7\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") "
Oct 30 03:17:24.029997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029852    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-data" pod "443f59af-9b47-43ea-91b1-63054fcd0ec7" (UID: "443f59af-9b47-43ea-91b1-63054fcd0ec7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:24.029997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029900    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-credential-socket\") pod \"443f59af-9b47-43ea-91b1-63054fcd0ec7\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") "
Oct 30 03:17:24.029997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.029938    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-kube-api-access-w6gg7\" (UniqueName: \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-kube-api-access-w6gg7\") pod \"443f59af-9b47-43ea-91b1-63054fcd0ec7\" (UID: \"443f59af-9b47-43ea-91b1-63054fcd0ec7\") "
Oct 30 03:17:24.030213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.030183    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-credential-socket" pod "443f59af-9b47-43ea-91b1-63054fcd0ec7" (UID: "443f59af-9b47-43ea-91b1-63054fcd0ec7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:24.030213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.030188    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443f59af-9b47-43ea-91b1-63054fcd0ec7-istiod-ca-cert" pod "443f59af-9b47-43ea-91b1-63054fcd0ec7" (UID: "443f59af-9b47-43ea-91b1-63054fcd0ec7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:24.030550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.030521    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-socket" pod "443f59af-9b47-43ea-91b1-63054fcd0ec7" (UID: "443f59af-9b47-43ea-91b1-63054fcd0ec7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:24.030883 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.030857    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:24.031014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.030886    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443f59af-9b47-43ea-91b1-63054fcd0ec7-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:24.031014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.030901    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:24.031014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.030916    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:24.031014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.030932    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:24.031994 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.031962    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-envoy" pod "443f59af-9b47-43ea-91b1-63054fcd0ec7" (UID: "443f59af-9b47-43ea-91b1-63054fcd0ec7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:24.032083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.032004    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-podinfo" pod "443f59af-9b47-43ea-91b1-63054fcd0ec7" (UID: "443f59af-9b47-43ea-91b1-63054fcd0ec7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:17:24.032083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.032025    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-token" pod "443f59af-9b47-43ea-91b1-63054fcd0ec7" (UID: "443f59af-9b47-43ea-91b1-63054fcd0ec7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:24.032283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.032262    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-kube-api-access-w6gg7" pod "443f59af-9b47-43ea-91b1-63054fcd0ec7" (UID: "443f59af-9b47-43ea-91b1-63054fcd0ec7"). InnerVolumeSpecName "kube-api-access-w6gg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:24.132029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.131989    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:24.132029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.132025    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:24.132252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.132038    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6gg7\" (UniqueName: \"kubernetes.io/projected/443f59af-9b47-43ea-91b1-63054fcd0ec7-kube-api-access-w6gg7\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:24.132252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.132054    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443f59af-9b47-43ea-91b1-63054fcd0ec7-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:24.192591 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:17:24.192532    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:17:24.255308 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.255271    4278 generic.go:356] "Generic (PLEG): container finished" podID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerID="6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712" exitCode=0
Oct 30 03:17:24.255538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.255326    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"
Oct 30 03:17:24.255538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.255352    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" event={"ID":"443f59af-9b47-43ea-91b1-63054fcd0ec7","Type":"ContainerDied","Data":"6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712"}
Oct 30 03:17:24.255538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.255412    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz" event={"ID":"443f59af-9b47-43ea-91b1-63054fcd0ec7","Type":"ContainerDied","Data":"ba4f2775eafed438ef6a4844c46fb7a4524817a7d2659136d5c16b36ef95a873"}
Oct 30 03:17:24.255538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.255472    4278 scope.go:122] "RemoveContainer" containerID="6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712"
Oct 30 03:17:24.269558 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.269538    4278 scope.go:122] "RemoveContainer" containerID="6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712"
Oct 30 03:17:24.269927 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:17:24.269904    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712\": container with ID starting with 6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712 not found: ID does not exist" containerID="6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712"
Oct 30 03:17:24.270002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.269939    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712"} err="failed to get container status \"6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712\": rpc error: code = NotFound desc = could not find container \"6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712\": container with ID starting with 6b153f3345402dd2ed257d1b891da7ced55e15ee4fbd613eb468ad65ebb78712 not found: ID does not exist"
Oct 30 03:17:24.314354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.314270    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"]
Oct 30 03:17:24.326893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:24.326860    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-p78pz"]
Oct 30 03:17:26.201825 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:26.201786    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" path="/var/lib/kubelet/pods/443f59af-9b47-43ea-91b1-63054fcd0ec7/volumes"
Oct 30 03:17:27.502008 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:27.501971    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"]
Oct 30 03:17:27.514659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:27.514617    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1fce3-8hlwt"]
Oct 30 03:17:28.201251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:28.201214    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72e37ed-cf22-44b0-a5f2-314b860d7f78" path="/var/lib/kubelet/pods/f72e37ed-cf22-44b0-a5f2-314b860d7f78/volumes"
Oct 30 03:17:30.984476 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:30.977256    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"]
Oct 30 03:17:31.490482 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.490438    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"]
Oct 30 03:17:31.491157 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.491141    4278 cpu_manager.go:414] "containerMap: removing container" podUID="2a39628b-20dd-4169-a2f5-672963350cfa" containerName="main"
Oct 30 03:17:31.491205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.491158    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a39628b-20dd-4169-a2f5-672963350cfa" containerName="main"
Oct 30 03:17:31.491205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.491172    4278 cpu_manager.go:414] "containerMap: removing container" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy"
Oct 30 03:17:31.491205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.491177    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy"
Oct 30 03:17:31.491305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.491296    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a39628b-20dd-4169-a2f5-672963350cfa" containerName="main"
Oct 30 03:17:31.491352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.491309    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="443f59af-9b47-43ea-91b1-63054fcd0ec7" containerName="istio-proxy"
Oct 30 03:17:31.495083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.495058    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:31.511684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.511656    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:17:31.511684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.511672    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:17:31.515496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.515463    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"]
Oct 30 03:17:31.566269 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.566226    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c653ed-7wrv5\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:31.566560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.566542    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxt86\" (UniqueName: \"kubernetes.io/projected/928e532c-1df8-4037-baa0-7773ad26a8c8-kube-api-access-zxt86\") pod \"maas-api-revoke-sub-e8c653ed-7wrv5\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:31.566674 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.566571    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c653ed-7wrv5\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:31.667988 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.667947    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxt86\" (UniqueName: \"kubernetes.io/projected/928e532c-1df8-4037-baa0-7773ad26a8c8-kube-api-access-zxt86\") pod \"maas-api-revoke-sub-e8c653ed-7wrv5\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:31.668192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.668006    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c653ed-7wrv5\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:31.668192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.668126    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c653ed-7wrv5\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:31.668913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.668881    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c653ed-7wrv5\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:31.670982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.670950    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c653ed-7wrv5\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:31.688998 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.688962    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxt86\" (UniqueName: \"kubernetes.io/projected/928e532c-1df8-4037-baa0-7773ad26a8c8-kube-api-access-zxt86\") pod \"maas-api-revoke-sub-e8c653ed-7wrv5\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:31.812348 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:31.812313    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:32.124621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:32.124586    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"]
Oct 30 03:17:32.125340 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:17:32.125315    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928e532c_1df8_4037_baa0_7773ad26a8c8.slice/crio-2ec935a55731b9e98292e251b341540398cc4bf8ef1085761d4ba928e129ce74 WatchSource:0}: Error finding container 2ec935a55731b9e98292e251b341540398cc4bf8ef1085761d4ba928e129ce74: Status 404 returned error can't find the container with id 2ec935a55731b9e98292e251b341540398cc4bf8ef1085761d4ba928e129ce74
Oct 30 03:17:32.327682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:32.327641    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5" event={"ID":"928e532c-1df8-4037-baa0-7773ad26a8c8","Type":"ContainerStarted","Data":"2ec935a55731b9e98292e251b341540398cc4bf8ef1085761d4ba928e129ce74"}
Oct 30 03:17:33.337044 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:33.337007    4278 generic.go:356] "Generic (PLEG): container finished" podID="928e532c-1df8-4037-baa0-7773ad26a8c8" containerID="401ffcfeea5c34587700440ad70a74875ecc1af409f388cee866a692effba10d" exitCode=0
Oct 30 03:17:33.337496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:33.337097    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5" event={"ID":"928e532c-1df8-4037-baa0-7773ad26a8c8","Type":"ContainerDied","Data":"401ffcfeea5c34587700440ad70a74875ecc1af409f388cee866a692effba10d"}
Oct 30 03:17:34.734213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.734184    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:34.838166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.838121    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-openshift-ca\") pod \"928e532c-1df8-4037-baa0-7773ad26a8c8\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") "
Oct 30 03:17:34.838351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.838257    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-xks-ca\") pod \"928e532c-1df8-4037-baa0-7773ad26a8c8\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") "
Oct 30 03:17:34.838493 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.838471    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928e532c-1df8-4037-baa0-7773ad26a8c8-kube-api-access-zxt86\" (UniqueName: \"kubernetes.io/projected/928e532c-1df8-4037-baa0-7773ad26a8c8-kube-api-access-zxt86\") pod \"928e532c-1df8-4037-baa0-7773ad26a8c8\" (UID: \"928e532c-1df8-4037-baa0-7773ad26a8c8\") "
Oct 30 03:17:34.841058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.841022    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-openshift-ca" pod "928e532c-1df8-4037-baa0-7773ad26a8c8" (UID: "928e532c-1df8-4037-baa0-7773ad26a8c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:34.843066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.843025    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928e532c-1df8-4037-baa0-7773ad26a8c8-kube-api-access-zxt86" pod "928e532c-1df8-4037-baa0-7773ad26a8c8" (UID: "928e532c-1df8-4037-baa0-7773ad26a8c8"). InnerVolumeSpecName "kube-api-access-zxt86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:34.843066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.843049    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-xks-ca" pod "928e532c-1df8-4037-baa0-7773ad26a8c8" (UID: "928e532c-1df8-4037-baa0-7773ad26a8c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:17:34.847147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.847115    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"]
Oct 30 03:17:34.847817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.847781    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy" containerID="cri-o://0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c" gracePeriod=30
Oct 30 03:17:34.950778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.950672    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:34.979019 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.978928    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928e532c-1df8-4037-baa0-7773ad26a8c8-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:34.979317 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:34.979296    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxt86\" (UniqueName: \"kubernetes.io/projected/928e532c-1df8-4037-baa0-7773ad26a8c8-kube-api-access-zxt86\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:35.353785 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.353752    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"
Oct 30 03:17:35.353977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.353751    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5" event={"ID":"928e532c-1df8-4037-baa0-7773ad26a8c8","Type":"ContainerDied","Data":"2ec935a55731b9e98292e251b341540398cc4bf8ef1085761d4ba928e129ce74"}
Oct 30 03:17:35.353977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.353878    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ec935a55731b9e98292e251b341540398cc4bf8ef1085761d4ba928e129ce74"
Oct 30 03:17:35.377863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.377827    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"]
Oct 30 03:17:35.378585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.378554    4278 cpu_manager.go:414] "containerMap: removing container" podUID="928e532c-1df8-4037-baa0-7773ad26a8c8" containerName="revoke-keys"
Oct 30 03:17:35.378585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.378583    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928e532c-1df8-4037-baa0-7773ad26a8c8" containerName="revoke-keys"
Oct 30 03:17:35.378752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.378734    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="928e532c-1df8-4037-baa0-7773ad26a8c8" containerName="revoke-keys"
Oct 30 03:17:35.382350 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.382333    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.402335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.402304    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"]
Oct 30 03:17:35.403272 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.403249    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:17:35.403401 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.403275    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:17:35.487282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.487243    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709ea3af-d601-430e-b475-7f468db73b39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.487556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.487300    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md7bt\" (UniqueName: \"kubernetes.io/projected/709ea3af-d601-430e-b475-7f468db73b39-kube-api-access-md7bt\") pod \"maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.487556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.487407    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709ea3af-d601-430e-b475-7f468db73b39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.590050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.590012    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709ea3af-d601-430e-b475-7f468db73b39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.590050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.590062    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7bt\" (UniqueName: \"kubernetes.io/projected/709ea3af-d601-430e-b475-7f468db73b39-kube-api-access-md7bt\") pod \"maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.590280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.590103    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709ea3af-d601-430e-b475-7f468db73b39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.590892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.590865    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709ea3af-d601-430e-b475-7f468db73b39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.592688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.592664    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709ea3af-d601-430e-b475-7f468db73b39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.610139 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.610078    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7bt\" (UniqueName: \"kubernetes.io/projected/709ea3af-d601-430e-b475-7f468db73b39-kube-api-access-md7bt\") pod \"maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.697294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.697257    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:35.767639 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.767604    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"]
Oct 30 03:17:35.777280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.776767    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:35.796563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.796531    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d3552\"/\"default-dockercfg-7fmhj\""
Oct 30 03:17:35.796964 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.796947    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d3552\"/\"openshift-service-ca.crt\""
Oct 30 03:17:35.797854 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.797684    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d3552\"/\"sub-test-model-9d3552-kserve-self-signed-certs\""
Oct 30 03:17:35.797854 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.797801    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d3552\"/\"kube-root-ca.crt\""
Oct 30 03:17:35.802124 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.802091    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"]
Oct 30 03:17:35.900223 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.900130    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-model-cache\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:35.900223 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.900195    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-dshm\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:35.900493 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.900271    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283ae180-cb3b-4819-96b4-66e394ccc791-tls-certs\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:35.900493 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.900342    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-home\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:35.900614 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.900507    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwp5f\" (UniqueName: \"kubernetes.io/projected/283ae180-cb3b-4819-96b4-66e394ccc791-kube-api-access-lwp5f\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:35.900672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:35.900650    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-tmp-dir\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.006451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.004683    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-home\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.006451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.004799    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwp5f\" (UniqueName: \"kubernetes.io/projected/283ae180-cb3b-4819-96b4-66e394ccc791-kube-api-access-lwp5f\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.006451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.004905    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-tmp-dir\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.006451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.005145    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-model-cache\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.006451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.005196    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-dshm\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.006451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.005225    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283ae180-cb3b-4819-96b4-66e394ccc791-tls-certs\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.006451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.005903    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-home\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.007060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.006533    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-tmp-dir\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.007060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.006778    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-model-cache\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.033448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.031079    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-dshm\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.052304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.052058    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"]
Oct 30 03:17:36.062978 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.062906    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwp5f\" (UniqueName: \"kubernetes.io/projected/283ae180-cb3b-4819-96b4-66e394ccc791-kube-api-access-lwp5f\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.090022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.082071    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283ae180-cb3b-4819-96b4-66e394ccc791-tls-certs\") pod \"sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.113091 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.101732    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:36.371755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.371683    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw" event={"ID":"709ea3af-d601-430e-b475-7f468db73b39","Type":"ContainerStarted","Data":"112d6b06b37f30dca25af8606df3124595fba9c1f25fbaca460ccbbda83b64c7"}
Oct 30 03:17:36.550224 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:36.548869    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"]
Oct 30 03:17:36.562668 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:17:36.562568    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod283ae180_cb3b_4819_96b4_66e394ccc791.slice/crio-a4f063eee4afe79043f8c69d41e8220b258e69bb44d0057a06d1870e27278de7 WatchSource:0}: Error finding container a4f063eee4afe79043f8c69d41e8220b258e69bb44d0057a06d1870e27278de7: Status 404 returned error can't find the container with id a4f063eee4afe79043f8c69d41e8220b258e69bb44d0057a06d1870e27278de7
Oct 30 03:17:37.382284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:37.382192    4278 generic.go:356] "Generic (PLEG): container finished" podID="709ea3af-d601-430e-b475-7f468db73b39" containerID="df994648d9dac2c3304e62deeb2db96daff94d4808f5c1cb33d0a5702904ad88" exitCode=0
Oct 30 03:17:37.382737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:37.382276    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw" event={"ID":"709ea3af-d601-430e-b475-7f468db73b39","Type":"ContainerDied","Data":"df994648d9dac2c3304e62deeb2db96daff94d4808f5c1cb33d0a5702904ad88"}
Oct 30 03:17:37.384118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:37.384083    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8" event={"ID":"283ae180-cb3b-4819-96b4-66e394ccc791","Type":"ContainerStarted","Data":"5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565"}
Oct 30 03:17:37.384231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:37.384136    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8" event={"ID":"283ae180-cb3b-4819-96b4-66e394ccc791","Type":"ContainerStarted","Data":"a4f063eee4afe79043f8c69d41e8220b258e69bb44d0057a06d1870e27278de7"}
Oct 30 03:17:37.461249 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:37.461179    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8" podStartSLOduration=2.461160953 podStartE2EDuration="2.461160953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:17:37.460539395 +0000 UTC m=+2195.384747589" watchObservedRunningTime="2026-10-30 03:17:37.461160953 +0000 UTC m=+2195.385369141"
Oct 30 03:17:38.102521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.102482    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:38.433836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.433792    4278 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:17:38.434451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.433868    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:17:38.582623 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.582593    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:38.756836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.756798    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709ea3af-d601-430e-b475-7f468db73b39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709ea3af-d601-430e-b475-7f468db73b39-maas-api-openshift-ca\") pod \"709ea3af-d601-430e-b475-7f468db73b39\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") "
Oct 30 03:17:38.757033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.756941    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709ea3af-d601-430e-b475-7f468db73b39-kube-api-access-md7bt\" (UniqueName: \"kubernetes.io/projected/709ea3af-d601-430e-b475-7f468db73b39-kube-api-access-md7bt\") pod \"709ea3af-d601-430e-b475-7f468db73b39\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") "
Oct 30 03:17:38.757132 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.757119    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/709ea3af-d601-430e-b475-7f468db73b39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709ea3af-d601-430e-b475-7f468db73b39-maas-api-xks-ca\") pod \"709ea3af-d601-430e-b475-7f468db73b39\" (UID: \"709ea3af-d601-430e-b475-7f468db73b39\") "
Oct 30 03:17:38.757197 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.757143    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709ea3af-d601-430e-b475-7f468db73b39-maas-api-openshift-ca" pod "709ea3af-d601-430e-b475-7f468db73b39" (UID: "709ea3af-d601-430e-b475-7f468db73b39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:38.757688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.757670    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709ea3af-d601-430e-b475-7f468db73b39-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:38.759219 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.759193    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709ea3af-d601-430e-b475-7f468db73b39-maas-api-xks-ca" pod "709ea3af-d601-430e-b475-7f468db73b39" (UID: "709ea3af-d601-430e-b475-7f468db73b39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:17:38.759300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.759280    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709ea3af-d601-430e-b475-7f468db73b39-kube-api-access-md7bt" pod "709ea3af-d601-430e-b475-7f468db73b39" (UID: "709ea3af-d601-430e-b475-7f468db73b39"). InnerVolumeSpecName "kube-api-access-md7bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:38.859021 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.858984    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md7bt\" (UniqueName: \"kubernetes.io/projected/709ea3af-d601-430e-b475-7f468db73b39-kube-api-access-md7bt\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:38.859021 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:38.859016    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709ea3af-d601-430e-b475-7f468db73b39-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:39.400559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:39.400519    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"
Oct 30 03:17:39.400559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:39.400540    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw" event={"ID":"709ea3af-d601-430e-b475-7f468db73b39","Type":"ContainerDied","Data":"112d6b06b37f30dca25af8606df3124595fba9c1f25fbaca460ccbbda83b64c7"}
Oct 30 03:17:39.400786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:39.400598    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112d6b06b37f30dca25af8606df3124595fba9c1f25fbaca460ccbbda83b64c7"
Oct 30 03:17:40.228965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.228927    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:40.385405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.385367    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-podinfo\") pod \"adcb3dd9-89f6-4a93-997f-dec09895c095\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") "
Oct 30 03:17:40.385630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.385589    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-token\" (UniqueName: \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-token\") pod \"adcb3dd9-89f6-4a93-997f-dec09895c095\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") "
Oct 30 03:17:40.385693 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.385678    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adcb3dd9-89f6-4a93-997f-dec09895c095-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcb3dd9-89f6-4a93-997f-dec09895c095-istiod-ca-cert\") pod \"adcb3dd9-89f6-4a93-997f-dec09895c095\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") "
Oct 30 03:17:40.385781 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.385738    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-kube-api-access-7sxzq\" (UniqueName: \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-kube-api-access-7sxzq\") pod \"adcb3dd9-89f6-4a93-997f-dec09895c095\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") "
Oct 30 03:17:40.385781 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.385768    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-data\") pod \"adcb3dd9-89f6-4a93-997f-dec09895c095\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") "
Oct 30 03:17:40.386099 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.386070    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-data" pod "adcb3dd9-89f6-4a93-997f-dec09895c095" (UID: "adcb3dd9-89f6-4a93-997f-dec09895c095"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.386220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.386097    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-credential-socket\") pod \"adcb3dd9-89f6-4a93-997f-dec09895c095\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") "
Oct 30 03:17:40.386220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.386153    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-certs\") pod \"adcb3dd9-89f6-4a93-997f-dec09895c095\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") "
Oct 30 03:17:40.386220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.386097    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcb3dd9-89f6-4a93-997f-dec09895c095-istiod-ca-cert" pod "adcb3dd9-89f6-4a93-997f-dec09895c095" (UID: "adcb3dd9-89f6-4a93-997f-dec09895c095"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:40.386220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.386210    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-socket\") pod \"adcb3dd9-89f6-4a93-997f-dec09895c095\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") "
Oct 30 03:17:40.386487 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.386308    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-credential-socket" pod "adcb3dd9-89f6-4a93-997f-dec09895c095" (UID: "adcb3dd9-89f6-4a93-997f-dec09895c095"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.386487 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.386351    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-envoy\") pod \"adcb3dd9-89f6-4a93-997f-dec09895c095\" (UID: \"adcb3dd9-89f6-4a93-997f-dec09895c095\") "
Oct 30 03:17:40.386614 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.386513    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-certs" pod "adcb3dd9-89f6-4a93-997f-dec09895c095" (UID: "adcb3dd9-89f6-4a93-997f-dec09895c095"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.386676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.386647    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-socket" pod "adcb3dd9-89f6-4a93-997f-dec09895c095" (UID: "adcb3dd9-89f6-4a93-997f-dec09895c095"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.387151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.387129    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcb3dd9-89f6-4a93-997f-dec09895c095-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.387151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.387153    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.387327 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.387165    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.387327 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.387181    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.387327 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.387193    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.388528 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.388311    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-kube-api-access-7sxzq" pod "adcb3dd9-89f6-4a93-997f-dec09895c095" (UID: "adcb3dd9-89f6-4a93-997f-dec09895c095"). InnerVolumeSpecName "kube-api-access-7sxzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:40.388851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.388824    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-podinfo" pod "adcb3dd9-89f6-4a93-997f-dec09895c095" (UID: "adcb3dd9-89f6-4a93-997f-dec09895c095"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:17:40.388948 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.388907    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-envoy" pod "adcb3dd9-89f6-4a93-997f-dec09895c095" (UID: "adcb3dd9-89f6-4a93-997f-dec09895c095"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.389168 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.389150    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-token" pod "adcb3dd9-89f6-4a93-997f-dec09895c095" (UID: "adcb3dd9-89f6-4a93-997f-dec09895c095"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:40.409237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.409205    4278 generic.go:356] "Generic (PLEG): container finished" podID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerID="0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c" exitCode=0
Oct 30 03:17:40.409343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.409269    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"
Oct 30 03:17:40.409343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.409289    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" event={"ID":"adcb3dd9-89f6-4a93-997f-dec09895c095","Type":"ContainerDied","Data":"0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c"}
Oct 30 03:17:40.409494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.409367    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9" event={"ID":"adcb3dd9-89f6-4a93-997f-dec09895c095","Type":"ContainerDied","Data":"8a078dc17a802d011e0f68c3a36bf4757243e304a4c8d20668f1ebf3030735f1"}
Oct 30 03:17:40.409494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.409397    4278 scope.go:122] "RemoveContainer" containerID="0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c"
Oct 30 03:17:40.429859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.429834    4278 scope.go:122] "RemoveContainer" containerID="0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c"
Oct 30 03:17:40.430220 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:17:40.430193    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c\": container with ID starting with 0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c not found: ID does not exist" containerID="0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c"
Oct 30 03:17:40.430291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.430240    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c"} err="failed to get container status \"0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c\": rpc error: code = NotFound desc = could not find container \"0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c\": container with ID starting with 0929b31c4d9daa5e98e5241c27bf78ed882f3c8fbd0aef098b1972bd2661a78c not found: ID does not exist"
Oct 30 03:17:40.467146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.467103    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"]
Oct 30 03:17:40.475699 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.475663    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-v9lc9"]
Oct 30 03:17:40.489135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.489095    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sxzq\" (UniqueName: \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-kube-api-access-7sxzq\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.489135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.489131    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.489393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.489147    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.489393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:40.489155    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcb3dd9-89f6-4a93-997f-dec09895c095-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:41.811748 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.811711    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"]
Oct 30 03:17:41.812416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.812372    4278 cpu_manager.go:414] "containerMap: removing container" podUID="709ea3af-d601-430e-b475-7f468db73b39" containerName="revoke-keys"
Oct 30 03:17:41.812519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.812418    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709ea3af-d601-430e-b475-7f468db73b39" containerName="revoke-keys"
Oct 30 03:17:41.812519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.812496    4278 cpu_manager.go:414] "containerMap: removing container" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy"
Oct 30 03:17:41.812519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.812504    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy"
Oct 30 03:17:41.812643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.812633    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" containerName="istio-proxy"
Oct 30 03:17:41.812677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.812648    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="709ea3af-d601-430e-b475-7f468db73b39" containerName="revoke-keys"
Oct 30 03:17:41.818336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.818312    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:41.845579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.843373    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"]
Oct 30 03:17:41.849323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.849291    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b0b40d\"/\"default-dockercfg-qwvz2\""
Oct 30 03:17:41.849323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.849310    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b0b40d\"/\"test-model-b0b40d-kserve-self-signed-certs\""
Oct 30 03:17:41.849569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.849352    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b0b40d\"/\"openshift-service-ca.crt\""
Oct 30 03:17:41.849569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.849372    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b0b40d\"/\"kube-root-ca.crt\""
Oct 30 03:17:41.915322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.915272    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-tmp-dir\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:41.915525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.915359    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-home\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:41.915525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.915511    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtswc\" (UniqueName: \"kubernetes.io/projected/349ef879-4d46-4762-a5ec-9bb20f4ac009-kube-api-access-jtswc\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:41.915688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.915666    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349ef879-4d46-4762-a5ec-9bb20f4ac009-tls-certs\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:41.915795 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.915775    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-model-cache\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:41.915859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:41.915815    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-dshm\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.017111 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.017071    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-tmp-dir\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.017286 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.017158    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-home\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.017286 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.017264    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtswc\" (UniqueName: \"kubernetes.io/projected/349ef879-4d46-4762-a5ec-9bb20f4ac009-kube-api-access-jtswc\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.017414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.017399    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349ef879-4d46-4762-a5ec-9bb20f4ac009-tls-certs\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.017530 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.017513    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-model-cache\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.017579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.017549    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-dshm\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.017669 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.017641    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-tmp-dir\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.017739 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.017686    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-home\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.017999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.017954    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-model-cache\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.020134 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.020102    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-dshm\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.020490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.020470    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349ef879-4d46-4762-a5ec-9bb20f4ac009-tls-certs\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.037418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.037382    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtswc\" (UniqueName: \"kubernetes.io/projected/349ef879-4d46-4762-a5ec-9bb20f4ac009-kube-api-access-jtswc\") pod \"test-model-b0b40d-kserve-7c8544d44c-l7s4q\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") " pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.140155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.139758    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:42.250414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.250383    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adcb3dd9-89f6-4a93-997f-dec09895c095" path="/var/lib/kubelet/pods/adcb3dd9-89f6-4a93-997f-dec09895c095/volumes"
Oct 30 03:17:42.544229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:42.544197    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"]
Oct 30 03:17:42.545246 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:17:42.545220    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod349ef879_4d46_4762_a5ec_9bb20f4ac009.slice/crio-153149249f183a0bcc00cb6c11ac1d706b36993d785247933f5c9ee8661d2adb WatchSource:0}: Error finding container 153149249f183a0bcc00cb6c11ac1d706b36993d785247933f5c9ee8661d2adb: Status 404 returned error can't find the container with id 153149249f183a0bcc00cb6c11ac1d706b36993d785247933f5c9ee8661d2adb
Oct 30 03:17:43.438841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:43.438800    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q" event={"ID":"349ef879-4d46-4762-a5ec-9bb20f4ac009","Type":"ContainerStarted","Data":"f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b"}
Oct 30 03:17:43.438841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:43.438852    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q" event={"ID":"349ef879-4d46-4762-a5ec-9bb20f4ac009","Type":"ContainerStarted","Data":"153149249f183a0bcc00cb6c11ac1d706b36993d785247933f5c9ee8661d2adb"}
Oct 30 03:17:43.482083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:43.482026    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q" podStartSLOduration=2.482008782 podStartE2EDuration="2.482008782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:17:43.478613667 +0000 UTC m=+2201.402821856" watchObservedRunningTime="2026-10-30 03:17:43.482008782 +0000 UTC m=+2201.406216969"
Oct 30 03:17:44.140971 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:44.140936    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:46.034101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.033994    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx" podUID="919c258f-20ff-488a-a62e-9bc61dcd5319" containerName="main" containerID="cri-o://f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27" gracePeriod=45
Oct 30 03:17:46.104395 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.103144    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:46.126484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.123180    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:46.439075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.439045    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:17:46.533516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.532293    4278 generic.go:356] "Generic (PLEG): container finished" podID="919c258f-20ff-488a-a62e-9bc61dcd5319" containerID="f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27" exitCode=0
Oct 30 03:17:46.535280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.534257    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx" event={"ID":"919c258f-20ff-488a-a62e-9bc61dcd5319","Type":"ContainerDied","Data":"f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27"}
Oct 30 03:17:46.535280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.534310    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx" event={"ID":"919c258f-20ff-488a-a62e-9bc61dcd5319","Type":"ContainerDied","Data":"bafd7ac7d13bf9c26341cb8d60b52e30e1afc1efdd38a6b6e8eb9d023f99a437"}
Oct 30 03:17:46.535280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.534336    4278 scope.go:122] "RemoveContainer" containerID="f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27"
Oct 30 03:17:46.535280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.534547    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"
Oct 30 03:17:46.739523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.736236    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/919c258f-20ff-488a-a62e-9bc61dcd5319-tls-certs\" (UniqueName: \"kubernetes.io/secret/919c258f-20ff-488a-a62e-9bc61dcd5319-tls-certs\") pod \"919c258f-20ff-488a-a62e-9bc61dcd5319\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") "
Oct 30 03:17:46.739523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.736404    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-home\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-home\") pod \"919c258f-20ff-488a-a62e-9bc61dcd5319\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") "
Oct 30 03:17:46.739523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.736674    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-model-cache\") pod \"919c258f-20ff-488a-a62e-9bc61dcd5319\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") "
Oct 30 03:17:46.739523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.736785    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-dshm\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-dshm\") pod \"919c258f-20ff-488a-a62e-9bc61dcd5319\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") "
Oct 30 03:17:46.739523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.736856    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919c258f-20ff-488a-a62e-9bc61dcd5319-kube-api-access-d6qzp\" (UniqueName: \"kubernetes.io/projected/919c258f-20ff-488a-a62e-9bc61dcd5319-kube-api-access-d6qzp\") pod \"919c258f-20ff-488a-a62e-9bc61dcd5319\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") "
Oct 30 03:17:46.739523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.737085    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-tmp-dir\") pod \"919c258f-20ff-488a-a62e-9bc61dcd5319\" (UID: \"919c258f-20ff-488a-a62e-9bc61dcd5319\") "
Oct 30 03:17:46.850142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.849539    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-home" pod "919c258f-20ff-488a-a62e-9bc61dcd5319" (UID: "919c258f-20ff-488a-a62e-9bc61dcd5319"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:46.852257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.850708    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-model-cache" pod "919c258f-20ff-488a-a62e-9bc61dcd5319" (UID: "919c258f-20ff-488a-a62e-9bc61dcd5319"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:46.855750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.854290    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-tmp-dir" pod "919c258f-20ff-488a-a62e-9bc61dcd5319" (UID: "919c258f-20ff-488a-a62e-9bc61dcd5319"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:46.860037 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.858588    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-dshm" pod "919c258f-20ff-488a-a62e-9bc61dcd5319" (UID: "919c258f-20ff-488a-a62e-9bc61dcd5319"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:46.863505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.860652    4278 scope.go:122] "RemoveContainer" containerID="f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27"
Oct 30 03:17:46.864084 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:17:46.864046    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27\": container with ID starting with f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27 not found: ID does not exist" containerID="f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27"
Oct 30 03:17:46.864202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.864116    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27"} err="failed to get container status \"f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27\": rpc error: code = NotFound desc = could not find container \"f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27\": container with ID starting with f6186d3088ebd64f8911d58d10b916e55944e2e2e350bbb6f2dfaa5780c80d27 not found: ID does not exist"
Oct 30 03:17:46.877531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.875824    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919c258f-20ff-488a-a62e-9bc61dcd5319-tls-certs" pod "919c258f-20ff-488a-a62e-9bc61dcd5319" (UID: "919c258f-20ff-488a-a62e-9bc61dcd5319"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:17:46.885775 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.885743    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919c258f-20ff-488a-a62e-9bc61dcd5319-kube-api-access-d6qzp" pod "919c258f-20ff-488a-a62e-9bc61dcd5319" (UID: "919c258f-20ff-488a-a62e-9bc61dcd5319"). InnerVolumeSpecName "kube-api-access-d6qzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:46.920858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.907568    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:17:46.956677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.956638    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:46.956677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.956684    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919c258f-20ff-488a-a62e-9bc61dcd5319-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:46.956946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.956698    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:46.956946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.956711    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:46.956946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.956723    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919c258f-20ff-488a-a62e-9bc61dcd5319-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:46.956946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:46.956741    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qzp\" (UniqueName: \"kubernetes.io/projected/919c258f-20ff-488a-a62e-9bc61dcd5319-kube-api-access-d6qzp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:47.010318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:47.010276    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"]
Oct 30 03:17:47.014188 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:47.014156    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-16df10/test-model-16df10-kserve-794dfff7bc-lj2sx"]
Oct 30 03:17:48.205861 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:48.205810    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919c258f-20ff-488a-a62e-9bc61dcd5319" path="/var/lib/kubelet/pods/919c258f-20ff-488a-a62e-9bc61dcd5319/volumes"
Oct 30 03:17:50.757041 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.757000    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"]
Oct 30 03:17:50.757680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.757665    4278 cpu_manager.go:414] "containerMap: removing container" podUID="919c258f-20ff-488a-a62e-9bc61dcd5319" containerName="main"
Oct 30 03:17:50.757737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.757681    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919c258f-20ff-488a-a62e-9bc61dcd5319" containerName="main"
Oct 30 03:17:50.757825 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.757816    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="919c258f-20ff-488a-a62e-9bc61dcd5319" containerName="main"
Oct 30 03:17:50.763167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.763143    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:50.782119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.782086    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"]
Oct 30 03:17:50.789179 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.789150    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:17:50.789344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.789204    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:17:50.832905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.830682    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21810f7d-pqjql\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:50.832905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.831017    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21810f7d-pqjql\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:50.832905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.831131    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klx96\" (UniqueName: \"kubernetes.io/projected/1381e3d4-a0a1-43d1-982e-2195686b533f-kube-api-access-klx96\") pod \"maas-api-revoke-sub-21810f7d-pqjql\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:50.933058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.933018    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21810f7d-pqjql\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:50.933058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.933060    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klx96\" (UniqueName: \"kubernetes.io/projected/1381e3d4-a0a1-43d1-982e-2195686b533f-kube-api-access-klx96\") pod \"maas-api-revoke-sub-21810f7d-pqjql\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:50.933341 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.933207    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21810f7d-pqjql\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:50.933743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.933719    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21810f7d-pqjql\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:50.935791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.935765    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21810f7d-pqjql\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:50.953313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:50.953275    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klx96\" (UniqueName: \"kubernetes.io/projected/1381e3d4-a0a1-43d1-982e-2195686b533f-kube-api-access-klx96\") pod \"maas-api-revoke-sub-21810f7d-pqjql\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:51.085122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:51.085028    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:51.400661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:51.400524    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"]
Oct 30 03:17:51.403276 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:17:51.401234    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1381e3d4_a0a1_43d1_982e_2195686b533f.slice/crio-9b334a97cd15104bcaeef5bfa7d45d2389f13cc410e06c0a4994e26ddb473b56 WatchSource:0}: Error finding container 9b334a97cd15104bcaeef5bfa7d45d2389f13cc410e06c0a4994e26ddb473b56: Status 404 returned error can't find the container with id 9b334a97cd15104bcaeef5bfa7d45d2389f13cc410e06c0a4994e26ddb473b56
Oct 30 03:17:51.577064 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:51.577032    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql" event={"ID":"1381e3d4-a0a1-43d1-982e-2195686b533f","Type":"ContainerStarted","Data":"9b334a97cd15104bcaeef5bfa7d45d2389f13cc410e06c0a4994e26ddb473b56"}
Oct 30 03:17:52.141454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:52.141330    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:52.154318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:52.154292    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:52.587036 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:52.586988    4278 generic.go:356] "Generic (PLEG): container finished" podID="1381e3d4-a0a1-43d1-982e-2195686b533f" containerID="37201fd8ed365b12b30f6251dbde89078a07872c5d8660a4acd9fc8d3b95eccb" exitCode=0
Oct 30 03:17:52.587255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:52.587071    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql" event={"ID":"1381e3d4-a0a1-43d1-982e-2195686b533f","Type":"ContainerDied","Data":"37201fd8ed365b12b30f6251dbde89078a07872c5d8660a4acd9fc8d3b95eccb"}
Oct 30 03:17:52.598809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:52.598772    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:17:53.593447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.592692    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"]
Oct 30 03:17:53.610501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.598741    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:53.622496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.614039    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"]
Oct 30 03:17:53.702843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.702806    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-726tv\" (UniqueName: \"kubernetes.io/projected/96ae4668-6eb5-4d76-8416-b1ea5a354076-kube-api-access-726tv\") pod \"maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:53.703095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.703011    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:53.703506 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.703468    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:53.805469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.805414    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:53.805645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.805585    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-726tv\" (UniqueName: \"kubernetes.io/projected/96ae4668-6eb5-4d76-8416-b1ea5a354076-kube-api-access-726tv\") pod \"maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:53.805717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.805644    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:53.806328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.806300    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:53.807992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.807968    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:53.825711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.825676    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-726tv\" (UniqueName: \"kubernetes.io/projected/96ae4668-6eb5-4d76-8416-b1ea5a354076-kube-api-access-726tv\") pod \"maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:53.854051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.853974    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:53.932576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:53.932537    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:54.008916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.008882    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1381e3d4-a0a1-43d1-982e-2195686b533f-kube-api-access-klx96\" (UniqueName: \"kubernetes.io/projected/1381e3d4-a0a1-43d1-982e-2195686b533f-kube-api-access-klx96\") pod \"1381e3d4-a0a1-43d1-982e-2195686b533f\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") "
Oct 30 03:17:54.009089 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.008933    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-xks-ca\") pod \"1381e3d4-a0a1-43d1-982e-2195686b533f\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") "
Oct 30 03:17:54.009089 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.008990    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-openshift-ca\") pod \"1381e3d4-a0a1-43d1-982e-2195686b533f\" (UID: \"1381e3d4-a0a1-43d1-982e-2195686b533f\") "
Oct 30 03:17:54.010789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.010753    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-openshift-ca" pod "1381e3d4-a0a1-43d1-982e-2195686b533f" (UID: "1381e3d4-a0a1-43d1-982e-2195686b533f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:54.012563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.012533    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-xks-ca" pod "1381e3d4-a0a1-43d1-982e-2195686b533f" (UID: "1381e3d4-a0a1-43d1-982e-2195686b533f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:17:54.013244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.013201    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1381e3d4-a0a1-43d1-982e-2195686b533f-kube-api-access-klx96" pod "1381e3d4-a0a1-43d1-982e-2195686b533f" (UID: "1381e3d4-a0a1-43d1-982e-2195686b533f"). InnerVolumeSpecName "kube-api-access-klx96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:54.111470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.111333    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klx96\" (UniqueName: \"kubernetes.io/projected/1381e3d4-a0a1-43d1-982e-2195686b533f-kube-api-access-klx96\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:54.111470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.111367    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:54.111470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.111377    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1381e3d4-a0a1-43d1-982e-2195686b533f-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:54.194670 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:17:54.194622    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:17:54.260521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.260483    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"]
Oct 30 03:17:54.260692 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:17:54.260665    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ae4668_6eb5_4d76_8416_b1ea5a354076.slice/crio-2cdca71a8abdda4d89fcecd79d5db4f1acf1769bcf296a0937bce466816c6ba9 WatchSource:0}: Error finding container 2cdca71a8abdda4d89fcecd79d5db4f1acf1769bcf296a0937bce466816c6ba9: Status 404 returned error can't find the container with id 2cdca71a8abdda4d89fcecd79d5db4f1acf1769bcf296a0937bce466816c6ba9
Oct 30 03:17:54.636703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.636670    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"
Oct 30 03:17:54.637197 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.636701    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql" event={"ID":"1381e3d4-a0a1-43d1-982e-2195686b533f","Type":"ContainerDied","Data":"9b334a97cd15104bcaeef5bfa7d45d2389f13cc410e06c0a4994e26ddb473b56"}
Oct 30 03:17:54.637197 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.636796    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b334a97cd15104bcaeef5bfa7d45d2389f13cc410e06c0a4994e26ddb473b56"
Oct 30 03:17:54.638510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:54.638479    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb" event={"ID":"96ae4668-6eb5-4d76-8416-b1ea5a354076","Type":"ContainerStarted","Data":"2cdca71a8abdda4d89fcecd79d5db4f1acf1769bcf296a0937bce466816c6ba9"}
Oct 30 03:17:55.647539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:55.647403    4278 generic.go:356] "Generic (PLEG): container finished" podID="96ae4668-6eb5-4d76-8416-b1ea5a354076" containerID="4c9e6f26381fa5e35ada310c1ae99cbd905a1ae7300d8e7b053094c1da0c4a13" exitCode=0
Oct 30 03:17:55.647539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:55.647498    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb" event={"ID":"96ae4668-6eb5-4d76-8416-b1ea5a354076","Type":"ContainerDied","Data":"4c9e6f26381fa5e35ada310c1ae99cbd905a1ae7300d8e7b053094c1da0c4a13"}
Oct 30 03:17:56.829402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:56.829370    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:56.867978 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:56.867921    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-openshift-ca\") pod \"96ae4668-6eb5-4d76-8416-b1ea5a354076\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") "
Oct 30 03:17:56.868225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:56.868068    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-xks-ca\") pod \"96ae4668-6eb5-4d76-8416-b1ea5a354076\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") "
Oct 30 03:17:56.868358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:56.868227    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ae4668-6eb5-4d76-8416-b1ea5a354076-kube-api-access-726tv\" (UniqueName: \"kubernetes.io/projected/96ae4668-6eb5-4d76-8416-b1ea5a354076-kube-api-access-726tv\") pod \"96ae4668-6eb5-4d76-8416-b1ea5a354076\" (UID: \"96ae4668-6eb5-4d76-8416-b1ea5a354076\") "
Oct 30 03:17:56.868846 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:56.868807    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-openshift-ca" pod "96ae4668-6eb5-4d76-8416-b1ea5a354076" (UID: "96ae4668-6eb5-4d76-8416-b1ea5a354076"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:56.869181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:56.869153    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:56.871679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:56.871650    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-xks-ca" pod "96ae4668-6eb5-4d76-8416-b1ea5a354076" (UID: "96ae4668-6eb5-4d76-8416-b1ea5a354076"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:17:56.871793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:56.871684    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ae4668-6eb5-4d76-8416-b1ea5a354076-kube-api-access-726tv" pod "96ae4668-6eb5-4d76-8416-b1ea5a354076" (UID: "96ae4668-6eb5-4d76-8416-b1ea5a354076"). InnerVolumeSpecName "kube-api-access-726tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:56.971039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:56.970950    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ae4668-6eb5-4d76-8416-b1ea5a354076-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:56.971039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:56.970984    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-726tv\" (UniqueName: \"kubernetes.io/projected/96ae4668-6eb5-4d76-8416-b1ea5a354076-kube-api-access-726tv\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:17:57.664519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:57.664482    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb" event={"ID":"96ae4668-6eb5-4d76-8416-b1ea5a354076","Type":"ContainerDied","Data":"2cdca71a8abdda4d89fcecd79d5db4f1acf1769bcf296a0937bce466816c6ba9"}
Oct 30 03:17:57.664519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:57.664513    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"
Oct 30 03:17:57.664760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:17:57.664537    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cdca71a8abdda4d89fcecd79d5db4f1acf1769bcf296a0937bce466816c6ba9"
Oct 30 03:18:01.151016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.150973    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"]
Oct 30 03:18:01.151763 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.151742    4278 cpu_manager.go:414] "containerMap: removing container" podUID="1381e3d4-a0a1-43d1-982e-2195686b533f" containerName="revoke-keys"
Oct 30 03:18:01.151823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.151763    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1381e3d4-a0a1-43d1-982e-2195686b533f" containerName="revoke-keys"
Oct 30 03:18:01.151823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.151792    4278 cpu_manager.go:414] "containerMap: removing container" podUID="96ae4668-6eb5-4d76-8416-b1ea5a354076" containerName="revoke-keys"
Oct 30 03:18:01.151823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.151799    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ae4668-6eb5-4d76-8416-b1ea5a354076" containerName="revoke-keys"
Oct 30 03:18:01.151927 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.151909    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="1381e3d4-a0a1-43d1-982e-2195686b533f" containerName="revoke-keys"
Oct 30 03:18:01.151927 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.151921    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ae4668-6eb5-4d76-8416-b1ea5a354076" containerName="revoke-keys"
Oct 30 03:18:01.155667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.155641    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.175840 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.175808    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"]
Oct 30 03:18:01.182982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.182933    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:18:01.183229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.183205    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:18:01.239130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.239088    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjklm\" (UniqueName: \"kubernetes.io/projected/8fd408d8-d117-45f7-8fc3-f2a1f220124f-kube-api-access-wjklm\") pod \"maas-api-revoke-sub-059b73c6-v9tp6\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.239326 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.239183    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059b73c6-v9tp6\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.239326 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.239294    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059b73c6-v9tp6\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.341357 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.341324    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjklm\" (UniqueName: \"kubernetes.io/projected/8fd408d8-d117-45f7-8fc3-f2a1f220124f-kube-api-access-wjklm\") pod \"maas-api-revoke-sub-059b73c6-v9tp6\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.341605 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.341413    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059b73c6-v9tp6\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.341605 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.341476    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059b73c6-v9tp6\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.342384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.342357    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059b73c6-v9tp6\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.343951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.343929    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059b73c6-v9tp6\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.362196 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.362168    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjklm\" (UniqueName: \"kubernetes.io/projected/8fd408d8-d117-45f7-8fc3-f2a1f220124f-kube-api-access-wjklm\") pod \"maas-api-revoke-sub-059b73c6-v9tp6\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.470676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.470576    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:01.786368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:01.786331    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"]
Oct 30 03:18:01.786615 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:01.786591    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd408d8_d117_45f7_8fc3_f2a1f220124f.slice/crio-554a24920b2bfb2377999cfa4ba9f2a98cdf2ba7423c7c776e9a01255cad228e WatchSource:0}: Error finding container 554a24920b2bfb2377999cfa4ba9f2a98cdf2ba7423c7c776e9a01255cad228e: Status 404 returned error can't find the container with id 554a24920b2bfb2377999cfa4ba9f2a98cdf2ba7423c7c776e9a01255cad228e
Oct 30 03:18:02.704660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:02.704616    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6" event={"ID":"8fd408d8-d117-45f7-8fc3-f2a1f220124f","Type":"ContainerStarted","Data":"5976fc20e905778e87789241174766b99d1cf081394ba8779fe4ff7a973c55f0"}
Oct 30 03:18:02.705158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:02.704684    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6" event={"ID":"8fd408d8-d117-45f7-8fc3-f2a1f220124f","Type":"ContainerStarted","Data":"554a24920b2bfb2377999cfa4ba9f2a98cdf2ba7423c7c776e9a01255cad228e"}
Oct 30 03:18:02.747056 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:02.746984    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6" podStartSLOduration=1.746960633 podStartE2EDuration="1.746960633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:02.742931257 +0000 UTC m=+2220.667139444" watchObservedRunningTime="2026-10-30 03:18:02.746960633 +0000 UTC m=+2220.671168824"
Oct 30 03:18:03.713696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:03.713657    4278 generic.go:356] "Generic (PLEG): container finished" podID="8fd408d8-d117-45f7-8fc3-f2a1f220124f" containerID="5976fc20e905778e87789241174766b99d1cf081394ba8779fe4ff7a973c55f0" exitCode=0
Oct 30 03:18:03.714208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:03.713764    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6" event={"ID":"8fd408d8-d117-45f7-8fc3-f2a1f220124f","Type":"ContainerDied","Data":"5976fc20e905778e87789241174766b99d1cf081394ba8779fe4ff7a973c55f0"}
Oct 30 03:18:03.999401 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:03.999302    4278 scope.go:122] "RemoveContainer" containerID="d1fc6b9567d22822ed102b5a8c30581e66ef80ae4f72898e0c48a50311efb95b"
Oct 30 03:18:04.010527 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.010503    4278 scope.go:122] "RemoveContainer" containerID="20031042400e856d446ee1bdcc53d4d2d275392dce13911974cda4afbca2d2a6"
Oct 30 03:18:04.458197 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.458159    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"]
Oct 30 03:18:04.464679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.464651    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.506737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.506704    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"]
Oct 30 03:18:04.507209 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.507176    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.507294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.507275    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.507542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.507520    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c223f4da-5498-444d-923d-63e48082c902-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.507665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.507593    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.507665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.507626    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c223f4da-5498-444d-923d-63e48082c902-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.507665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.507651    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.507803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.507794    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.508072 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.507997    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2tw\" (UniqueName: \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-kube-api-access-sh2tw\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.508072 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.508040    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.609994 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.609955    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610043    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2tw\" (UniqueName: \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-kube-api-access-sh2tw\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610075    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610142    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610214    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610329    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c223f4da-5498-444d-923d-63e48082c902-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610395    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610447    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c223f4da-5498-444d-923d-63e48082c902-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610576    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610621    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610635    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610679    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.610954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.610926    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.611100 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.611080    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c223f4da-5498-444d-923d-63e48082c902-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.612573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.612549    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.612898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.612878    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c223f4da-5498-444d-923d-63e48082c902-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.630247 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.630186    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.630407 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.630390    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2tw\" (UniqueName: \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-kube-api-access-sh2tw\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:04.810791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:04.800823    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:05.076159 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.076130    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:05.163296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.163253    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd408d8-d117-45f7-8fc3-f2a1f220124f-kube-api-access-wjklm\" (UniqueName: \"kubernetes.io/projected/8fd408d8-d117-45f7-8fc3-f2a1f220124f-kube-api-access-wjklm\") pod \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") "
Oct 30 03:18:05.163548 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.163446    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-xks-ca\") pod \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") "
Oct 30 03:18:05.163688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.163639    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-openshift-ca\") pod \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\" (UID: \"8fd408d8-d117-45f7-8fc3-f2a1f220124f\") "
Oct 30 03:18:05.164464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.164202    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-openshift-ca" pod "8fd408d8-d117-45f7-8fc3-f2a1f220124f" (UID: "8fd408d8-d117-45f7-8fc3-f2a1f220124f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:05.164464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.164415    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:05.166331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.166295    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd408d8-d117-45f7-8fc3-f2a1f220124f-kube-api-access-wjklm" pod "8fd408d8-d117-45f7-8fc3-f2a1f220124f" (UID: "8fd408d8-d117-45f7-8fc3-f2a1f220124f"). InnerVolumeSpecName "kube-api-access-wjklm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:05.166483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.166391    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-xks-ca" pod "8fd408d8-d117-45f7-8fc3-f2a1f220124f" (UID: "8fd408d8-d117-45f7-8fc3-f2a1f220124f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:05.196764 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:05.196736    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc223f4da_5498_444d_923d_63e48082c902.slice/crio-afc24a72e9ec8f8a8e9e445ff4039099487d365deae146a3498856d1b61479da WatchSource:0}: Error finding container afc24a72e9ec8f8a8e9e445ff4039099487d365deae146a3498856d1b61479da: Status 404 returned error can't find the container with id afc24a72e9ec8f8a8e9e445ff4039099487d365deae146a3498856d1b61479da
Oct 30 03:18:05.196941 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.196923    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"]
Oct 30 03:18:05.200657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.200623    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:05.200758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.200692    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:05.200758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.200722    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:05.266332 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.266298    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjklm\" (UniqueName: \"kubernetes.io/projected/8fd408d8-d117-45f7-8fc3-f2a1f220124f-kube-api-access-wjklm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:05.266516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.266336    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd408d8-d117-45f7-8fc3-f2a1f220124f-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:05.864901 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.864871    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"
Oct 30 03:18:05.865408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.864911    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6" event={"ID":"8fd408d8-d117-45f7-8fc3-f2a1f220124f","Type":"ContainerDied","Data":"554a24920b2bfb2377999cfa4ba9f2a98cdf2ba7423c7c776e9a01255cad228e"}
Oct 30 03:18:05.865408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.865002    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="554a24920b2bfb2377999cfa4ba9f2a98cdf2ba7423c7c776e9a01255cad228e"
Oct 30 03:18:05.867744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.867708    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" event={"ID":"c223f4da-5498-444d-923d-63e48082c902","Type":"ContainerStarted","Data":"bfe4bc1548fa5afe0a187b42ebae4f8a6da7824a3cf995bf9b2447807354c0d9"}
Oct 30 03:18:05.867928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.867767    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" event={"ID":"c223f4da-5498-444d-923d-63e48082c902","Type":"ContainerStarted","Data":"afc24a72e9ec8f8a8e9e445ff4039099487d365deae146a3498856d1b61479da"}
Oct 30 03:18:05.910310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:05.910082    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" podStartSLOduration=1.9100617560000002 podStartE2EDuration="1.910061756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:05.91003434 +0000 UTC m=+2223.834242541" watchObservedRunningTime="2026-10-30 03:18:05.910061756 +0000 UTC m=+2223.834269940"
Oct 30 03:18:06.801154 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:06.801120    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:06.803038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:06.803005    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 03:18:06.803178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:06.803060    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 03:18:07.801728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:07.801693    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 03:18:07.802161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:07.801750    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 03:18:08.801700 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:08.801658    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 03:18:08.801897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:08.801733    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 03:18:09.802002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.801958    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 03:18:09.802623 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.802049    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 03:18:09.851968 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.851928    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"]
Oct 30 03:18:09.852628 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.852612    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8fd408d8-d117-45f7-8fc3-f2a1f220124f" containerName="revoke-keys"
Oct 30 03:18:09.852697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.852629    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd408d8-d117-45f7-8fc3-f2a1f220124f" containerName="revoke-keys"
Oct 30 03:18:09.852769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.852760    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd408d8-d117-45f7-8fc3-f2a1f220124f" containerName="revoke-keys"
Oct 30 03:18:09.856313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.856293    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:09.887867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.878676    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:18:09.887867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.887335    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:18:09.891600 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.891555    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"]
Oct 30 03:18:09.984577 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.984512    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21288bcc-m5b8h\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:09.984820 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.984785    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21288bcc-m5b8h\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:09.985084 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:09.985058    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx92l\" (UniqueName: \"kubernetes.io/projected/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-kube-api-access-lx92l\") pod \"maas-api-revoke-sub-21288bcc-m5b8h\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:10.086591 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.086481    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx92l\" (UniqueName: \"kubernetes.io/projected/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-kube-api-access-lx92l\") pod \"maas-api-revoke-sub-21288bcc-m5b8h\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:10.086788 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.086631    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21288bcc-m5b8h\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:10.086862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.086836    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21288bcc-m5b8h\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:10.087668 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.087633    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21288bcc-m5b8h\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:10.089259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.089235    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21288bcc-m5b8h\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:10.106571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.106540    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx92l\" (UniqueName: \"kubernetes.io/projected/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-kube-api-access-lx92l\") pod \"maas-api-revoke-sub-21288bcc-m5b8h\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:10.206111 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.206078    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:10.544408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.544366    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"]
Oct 30 03:18:10.545561 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:10.545526    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedfdb3a_4015_4ba4_b6a7_3b2558e9a183.slice/crio-7b9e9299e9a3b3366131c12edcf6c07a239d43d0cdce6639d51c60dc6d63cfeb WatchSource:0}: Error finding container 7b9e9299e9a3b3366131c12edcf6c07a239d43d0cdce6639d51c60dc6d63cfeb: Status 404 returned error can't find the container with id 7b9e9299e9a3b3366131c12edcf6c07a239d43d0cdce6639d51c60dc6d63cfeb
Oct 30 03:18:10.802314 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.802224    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 03:18:10.802314 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.802302    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 03:18:10.850129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.850088    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"]
Oct 30 03:18:10.855990 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.855963    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:10.886764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.885647    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"]
Oct 30 03:18:10.932139 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.932103    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h" event={"ID":"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183","Type":"ContainerStarted","Data":"7b9e9299e9a3b3366131c12edcf6c07a239d43d0cdce6639d51c60dc6d63cfeb"}
Oct 30 03:18:10.973448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.970989    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"]
Oct 30 03:18:10.981627 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.980788    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:10.999517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.996887    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-100ffd-openshift-default-dockercfg-db8dw\""
Oct 30 03:18:10.999517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:10.998899    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"]
Oct 30 03:18:11.015714 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.015676    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbshw\" (UniqueName: \"kubernetes.io/projected/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-kube-api-access-sbshw\") pod \"maas-api-revoke-sub-8104b9fc-rztgt\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:11.015914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.015760    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8104b9fc-rztgt\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:11.015914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.015871    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8104b9fc-rztgt\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:11.117784 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.117745    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istiod-ca-cert\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.117995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.117895    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-podinfo\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.117995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.117989    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-certs\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.118084 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.118034    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbshw\" (UniqueName: \"kubernetes.io/projected/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-kube-api-access-sbshw\") pod \"maas-api-revoke-sub-8104b9fc-rztgt\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:11.118136 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.118095    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8104b9fc-rztgt\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:11.118206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.118175    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfl5t\" (UniqueName: \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-kube-api-access-qfl5t\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.118265 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.118220    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-envoy\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.118321 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.118260    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-credential-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.118375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.118331    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.118445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.118381    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-data\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.118522 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.118507    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8104b9fc-rztgt\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:11.118670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.118651    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-token\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.119164 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.119137    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8104b9fc-rztgt\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:11.120982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.120960    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8104b9fc-rztgt\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:11.138233 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.138195    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbshw\" (UniqueName: \"kubernetes.io/projected/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-kube-api-access-sbshw\") pod \"maas-api-revoke-sub-8104b9fc-rztgt\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:11.216170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.216129    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:11.220049 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.219850    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istiod-ca-cert\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220049 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.219935    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-podinfo\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220049 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.219988    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-certs\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.220105    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfl5t\" (UniqueName: \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-kube-api-access-qfl5t\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.220138    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-envoy\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.220168    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-credential-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.220230    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.220279    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-data\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.220321    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-token\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220597 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.220533    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-credential-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220827 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.220803    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.220908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.220860    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-data\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.221092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.221066    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istiod-ca-cert\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.221224 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.221194    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-certs\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.223107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.223083    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-envoy\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.223539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.223517    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-podinfo\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.241196 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.241135    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-token\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.241489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.241413    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfl5t\" (UniqueName: \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-kube-api-access-qfl5t\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.307736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.307693    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:11.598409 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.597944    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"]
Oct 30 03:18:11.602806 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:11.601586    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71743ac3_2f89_4cf3_9f83_42f7ad51fe05.slice/crio-e4ef19152d83f441687cdddb1f23d43fec2b7b73308674986a05cc8e4c0858a3 WatchSource:0}: Error finding container e4ef19152d83f441687cdddb1f23d43fec2b7b73308674986a05cc8e4c0858a3: Status 404 returned error can't find the container with id e4ef19152d83f441687cdddb1f23d43fec2b7b73308674986a05cc8e4c0858a3
Oct 30 03:18:11.702266 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.702230    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"]
Oct 30 03:18:11.762957 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:11.762912    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da52699_c3f3_420e_9db9_b52af3d7cc6f.slice/crio-42d6aa4899893b2a124144ac36edd38a683b33223ba2d93a07927fb4112b99a2 WatchSource:0}: Error finding container 42d6aa4899893b2a124144ac36edd38a683b33223ba2d93a07927fb4112b99a2: Status 404 returned error can't find the container with id 42d6aa4899893b2a124144ac36edd38a683b33223ba2d93a07927fb4112b99a2
Oct 30 03:18:11.766289 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.766240    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:11.766411 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.766328    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:11.766411 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.766359    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:11.801535 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.801505    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 03:18:11.801681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.801567    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 03:18:11.946279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.946232    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" event={"ID":"5da52699-c3f3-420e-9db9-b52af3d7cc6f","Type":"ContainerStarted","Data":"65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704"}
Oct 30 03:18:11.946967 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.946935    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" event={"ID":"5da52699-c3f3-420e-9db9-b52af3d7cc6f","Type":"ContainerStarted","Data":"42d6aa4899893b2a124144ac36edd38a683b33223ba2d93a07927fb4112b99a2"}
Oct 30 03:18:11.948144 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.948113    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt" event={"ID":"71743ac3-2f89-4cf3-9f83-42f7ad51fe05","Type":"ContainerStarted","Data":"e4ef19152d83f441687cdddb1f23d43fec2b7b73308674986a05cc8e4c0858a3"}
Oct 30 03:18:11.950374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.950349    4278 generic.go:356] "Generic (PLEG): container finished" podID="cedfdb3a-4015-4ba4-b6a7-3b2558e9a183" containerID="1715f8d551511144ab269a8ae32652c27314f7211a2fef3c0a65e78715dbca98" exitCode=0
Oct 30 03:18:11.950547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.950406    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h" event={"ID":"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183","Type":"ContainerDied","Data":"1715f8d551511144ab269a8ae32652c27314f7211a2fef3c0a65e78715dbca98"}
Oct 30 03:18:11.994778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:11.994710    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" podStartSLOduration=1.9946934600000001 podStartE2EDuration="1.99469346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:11.989288464 +0000 UTC m=+2229.913496654" watchObservedRunningTime="2026-10-30 03:18:11.99469346 +0000 UTC m=+2229.918901737"
Oct 30 03:18:12.010659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.010622    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"]
Oct 30 03:18:12.016272 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.016242    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.035371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.035338    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"]
Oct 30 03:18:12.150783 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.150726    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxkb\" (UniqueName: \"kubernetes.io/projected/f8842ecd-e09b-455e-86dd-d31018e7ef55-kube-api-access-8fxkb\") pod \"maas-api-revoke-sub-4811b439-42fth\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.151006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.150835    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4811b439-42fth\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.151006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.150903    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4811b439-42fth\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.253260 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.253214    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4811b439-42fth\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.253498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.253323    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4811b439-42fth\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.253768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.253741    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxkb\" (UniqueName: \"kubernetes.io/projected/f8842ecd-e09b-455e-86dd-d31018e7ef55-kube-api-access-8fxkb\") pod \"maas-api-revoke-sub-4811b439-42fth\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.254147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.254127    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4811b439-42fth\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.256463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.256402    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4811b439-42fth\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.274850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.274810    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxkb\" (UniqueName: \"kubernetes.io/projected/f8842ecd-e09b-455e-86dd-d31018e7ef55-kube-api-access-8fxkb\") pod \"maas-api-revoke-sub-4811b439-42fth\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.309033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.308878    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:12.310930 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.310894    4278 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:18:12.311095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.310968    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 03:18:12.339727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.339676    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:12.696166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.696133    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"]
Oct 30 03:18:12.807291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.807259    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:12.807698 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.807667    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:12.808497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:12.808477    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:18:13.002034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.001995    4278 generic.go:356] "Generic (PLEG): container finished" podID="71743ac3-2f89-4cf3-9f83-42f7ad51fe05" containerID="c2f3e7eaa178e48fd604632bbd5cfc2c69718e3c767e7905889e49506114db2e" exitCode=0
Oct 30 03:18:13.002453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.002095    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt" event={"ID":"71743ac3-2f89-4cf3-9f83-42f7ad51fe05","Type":"ContainerDied","Data":"c2f3e7eaa178e48fd604632bbd5cfc2c69718e3c767e7905889e49506114db2e"}
Oct 30 03:18:13.005545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.005511    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth" event={"ID":"f8842ecd-e09b-455e-86dd-d31018e7ef55","Type":"ContainerStarted","Data":"10d558d125538ebdf1e057e279112b0a60969d44692dc93da2ada17dab06745e"}
Oct 30 03:18:13.240978 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.240950    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:13.308601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.308558    4278 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:18:13.308815 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.308625    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 03:18:13.391244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.391197    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-xks-ca\") pod \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") "
Oct 30 03:18:13.391244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.391242    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-kube-api-access-lx92l\" (UniqueName: \"kubernetes.io/projected/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-kube-api-access-lx92l\") pod \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") "
Oct 30 03:18:13.391580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.391563    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-openshift-ca\") pod \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\" (UID: \"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183\") "
Oct 30 03:18:13.391889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.391859    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-openshift-ca" pod "cedfdb3a-4015-4ba4-b6a7-3b2558e9a183" (UID: "cedfdb3a-4015-4ba4-b6a7-3b2558e9a183"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:13.392361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.392339    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:13.393601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.393569    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-xks-ca" pod "cedfdb3a-4015-4ba4-b6a7-3b2558e9a183" (UID: "cedfdb3a-4015-4ba4-b6a7-3b2558e9a183"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:13.393695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.393652    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-kube-api-access-lx92l" pod "cedfdb3a-4015-4ba4-b6a7-3b2558e9a183" (UID: "cedfdb3a-4015-4ba4-b6a7-3b2558e9a183"). InnerVolumeSpecName "kube-api-access-lx92l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:13.494170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.494068    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:13.494170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:13.494100    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx92l\" (UniqueName: \"kubernetes.io/projected/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183-kube-api-access-lx92l\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:14.122830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.121547    4278 generic.go:356] "Generic (PLEG): container finished" podID="f8842ecd-e09b-455e-86dd-d31018e7ef55" containerID="f6f1d2affe7b4287fd40a080da9ef419189e9a1603cdf26790ab34cc96ff0cc5" exitCode=0
Oct 30 03:18:14.122830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.122154    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth" event={"ID":"f8842ecd-e09b-455e-86dd-d31018e7ef55","Type":"ContainerDied","Data":"f6f1d2affe7b4287fd40a080da9ef419189e9a1603cdf26790ab34cc96ff0cc5"}
Oct 30 03:18:14.124680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.124654    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"
Oct 30 03:18:14.125765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.125743    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h" event={"ID":"cedfdb3a-4015-4ba4-b6a7-3b2558e9a183","Type":"ContainerDied","Data":"7b9e9299e9a3b3366131c12edcf6c07a239d43d0cdce6639d51c60dc6d63cfeb"}
Oct 30 03:18:14.125840 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.125782    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b9e9299e9a3b3366131c12edcf6c07a239d43d0cdce6639d51c60dc6d63cfeb"
Oct 30 03:18:14.308722 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.308527    4278 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:18:14.308722 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.308636    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 03:18:14.356151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.356125    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:14.511030 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.510994    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-openshift-ca\") pod \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") "
Oct 30 03:18:14.511255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.511048    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-kube-api-access-sbshw\" (UniqueName: \"kubernetes.io/projected/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-kube-api-access-sbshw\") pod \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") "
Oct 30 03:18:14.511255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.511175    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-xks-ca\") pod \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\" (UID: \"71743ac3-2f89-4cf3-9f83-42f7ad51fe05\") "
Oct 30 03:18:14.511494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.511458    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-openshift-ca" pod "71743ac3-2f89-4cf3-9f83-42f7ad51fe05" (UID: "71743ac3-2f89-4cf3-9f83-42f7ad51fe05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:14.511847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.511826    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:14.513312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.513281    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-xks-ca" pod "71743ac3-2f89-4cf3-9f83-42f7ad51fe05" (UID: "71743ac3-2f89-4cf3-9f83-42f7ad51fe05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:14.513920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.513320    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-kube-api-access-sbshw" pod "71743ac3-2f89-4cf3-9f83-42f7ad51fe05" (UID: "71743ac3-2f89-4cf3-9f83-42f7ad51fe05"). InnerVolumeSpecName "kube-api-access-sbshw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:14.613052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.612961    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:14.613052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:14.612991    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbshw\" (UniqueName: \"kubernetes.io/projected/71743ac3-2f89-4cf3-9f83-42f7ad51fe05-kube-api-access-sbshw\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:15.133892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.133845    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"
Oct 30 03:18:15.133892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.133871    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt" event={"ID":"71743ac3-2f89-4cf3-9f83-42f7ad51fe05","Type":"ContainerDied","Data":"e4ef19152d83f441687cdddb1f23d43fec2b7b73308674986a05cc8e4c0858a3"}
Oct 30 03:18:15.134466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.133926    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4ef19152d83f441687cdddb1f23d43fec2b7b73308674986a05cc8e4c0858a3"
Oct 30 03:18:15.310059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.309995    4278 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:18:15.310279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.310096    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 03:18:15.453630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.453592    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:15.547472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.545415    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-openshift-ca\") pod \"f8842ecd-e09b-455e-86dd-d31018e7ef55\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") "
Oct 30 03:18:15.547472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.545525    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-xks-ca\") pod \"f8842ecd-e09b-455e-86dd-d31018e7ef55\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") "
Oct 30 03:18:15.547472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.545567    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8842ecd-e09b-455e-86dd-d31018e7ef55-kube-api-access-8fxkb\" (UniqueName: \"kubernetes.io/projected/f8842ecd-e09b-455e-86dd-d31018e7ef55-kube-api-access-8fxkb\") pod \"f8842ecd-e09b-455e-86dd-d31018e7ef55\" (UID: \"f8842ecd-e09b-455e-86dd-d31018e7ef55\") "
Oct 30 03:18:15.766516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.707625    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8842ecd-e09b-455e-86dd-d31018e7ef55-kube-api-access-8fxkb" pod "f8842ecd-e09b-455e-86dd-d31018e7ef55" (UID: "f8842ecd-e09b-455e-86dd-d31018e7ef55"). InnerVolumeSpecName "kube-api-access-8fxkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:15.801598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.787038    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-openshift-ca" pod "f8842ecd-e09b-455e-86dd-d31018e7ef55" (UID: "f8842ecd-e09b-455e-86dd-d31018e7ef55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:15.801598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.789742    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:15.801598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.789771    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxkb\" (UniqueName: \"kubernetes.io/projected/f8842ecd-e09b-455e-86dd-d31018e7ef55-kube-api-access-8fxkb\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:15.972445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:15.969302    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-xks-ca" pod "f8842ecd-e09b-455e-86dd-d31018e7ef55" (UID: "f8842ecd-e09b-455e-86dd-d31018e7ef55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:16.058513 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:16.055061    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8842ecd-e09b-455e-86dd-d31018e7ef55-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:16.148685 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:16.148647    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"
Oct 30 03:18:16.148685 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:16.148669    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth" event={"ID":"f8842ecd-e09b-455e-86dd-d31018e7ef55","Type":"ContainerDied","Data":"10d558d125538ebdf1e057e279112b0a60969d44692dc93da2ada17dab06745e"}
Oct 30 03:18:16.149289 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:16.148733    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10d558d125538ebdf1e057e279112b0a60969d44692dc93da2ada17dab06745e"
Oct 30 03:18:16.308168 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:16.308132    4278 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:18:16.308365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:16.308196    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 03:18:16.329359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:16.329266    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"]
Oct 30 03:18:16.333420 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:16.333393    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f415d-59zrw"]
Oct 30 03:18:17.308742 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:17.308705    4278 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:18:17.309175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:17.308759    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 03:18:18.203105 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:18.203064    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709ea3af-d601-430e-b475-7f468db73b39" path="/var/lib/kubelet/pods/709ea3af-d601-430e-b475-7f468db73b39/volumes"
Oct 30 03:18:18.311857 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:18.311811    4278 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:18:18.312917 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:18.312501    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 03:18:19.313753 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:19.313721    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:19.314250 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:19.314191    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:19.315035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:19.315010    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:18:20.662226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:20.662188    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:18:22.574490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.574447    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"]
Oct 30 03:18:22.575178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.575159    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f8842ecd-e09b-455e-86dd-d31018e7ef55" containerName="revoke-keys"
Oct 30 03:18:22.575259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.575180    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8842ecd-e09b-455e-86dd-d31018e7ef55" containerName="revoke-keys"
Oct 30 03:18:22.575259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.575227    4278 cpu_manager.go:414] "containerMap: removing container" podUID="cedfdb3a-4015-4ba4-b6a7-3b2558e9a183" containerName="revoke-keys"
Oct 30 03:18:22.575259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.575234    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cedfdb3a-4015-4ba4-b6a7-3b2558e9a183" containerName="revoke-keys"
Oct 30 03:18:22.575414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.575263    4278 cpu_manager.go:414] "containerMap: removing container" podUID="71743ac3-2f89-4cf3-9f83-42f7ad51fe05" containerName="revoke-keys"
Oct 30 03:18:22.575414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.575269    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71743ac3-2f89-4cf3-9f83-42f7ad51fe05" containerName="revoke-keys"
Oct 30 03:18:22.575414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.575383    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="cedfdb3a-4015-4ba4-b6a7-3b2558e9a183" containerName="revoke-keys"
Oct 30 03:18:22.575414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.575398    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="71743ac3-2f89-4cf3-9f83-42f7ad51fe05" containerName="revoke-keys"
Oct 30 03:18:22.575414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.575411    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8842ecd-e09b-455e-86dd-d31018e7ef55" containerName="revoke-keys"
Oct 30 03:18:22.581225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.581199    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:22.601101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.601067    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"]
Oct 30 03:18:22.607331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.607301    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:18:22.607677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.607633    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:18:22.787798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.787758    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a42311f3-z2klm\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:22.788156 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.788117    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a42311f3-z2klm\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:22.788615 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.788593    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsd82\" (UniqueName: \"kubernetes.io/projected/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-kube-api-access-vsd82\") pod \"maas-api-revoke-sub-a42311f3-z2klm\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:22.948506 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.933401    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsd82\" (UniqueName: \"kubernetes.io/projected/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-kube-api-access-vsd82\") pod \"maas-api-revoke-sub-a42311f3-z2klm\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:22.949571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.949532    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a42311f3-z2klm\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:22.950295 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.950271    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a42311f3-z2klm\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:22.955501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:22.951169    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a42311f3-z2klm\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:23.019861 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:23.019787    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a42311f3-z2klm\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:23.042449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:23.038183    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsd82\" (UniqueName: \"kubernetes.io/projected/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-kube-api-access-vsd82\") pod \"maas-api-revoke-sub-a42311f3-z2klm\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:23.205894 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:23.205807    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:23.530469 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:23.530388    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c1f6f5_1d80_4bb3_bf49_4f2e4feac6ba.slice/crio-42fc4d5cf148fbf801ed131183d4289cfd525922e6c43645a4296aa6e9870c70 WatchSource:0}: Error finding container 42fc4d5cf148fbf801ed131183d4289cfd525922e6c43645a4296aa6e9870c70: Status 404 returned error can't find the container with id 42fc4d5cf148fbf801ed131183d4289cfd525922e6c43645a4296aa6e9870c70
Oct 30 03:18:23.530879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:23.530848    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"]
Oct 30 03:18:24.232027 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:18:24.230876    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:18:24.234320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:24.234288    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"]
Oct 30 03:18:24.234601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:24.234573    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy" containerID="cri-o://baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93" gracePeriod=30
Oct 30 03:18:24.240915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:24.240886    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm" event={"ID":"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba","Type":"ContainerStarted","Data":"6e9863f26011b048cad576d856020723659a577fae0809fa877c08d06939c1a5"}
Oct 30 03:18:24.241094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:24.240965    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm" event={"ID":"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba","Type":"ContainerStarted","Data":"42fc4d5cf148fbf801ed131183d4289cfd525922e6c43645a4296aa6e9870c70"}
Oct 30 03:18:24.382645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:24.374054    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm" podStartSLOduration=2.374030142 podStartE2EDuration="2.374030142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:24.362449606 +0000 UTC m=+2242.286657793" watchObservedRunningTime="2026-10-30 03:18:24.374030142 +0000 UTC m=+2242.298238332"
Oct 30 03:18:25.253517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:25.253482    4278 generic.go:356] "Generic (PLEG): container finished" podID="71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba" containerID="6e9863f26011b048cad576d856020723659a577fae0809fa877c08d06939c1a5" exitCode=0
Oct 30 03:18:25.253958 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:25.253563    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm" event={"ID":"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba","Type":"ContainerDied","Data":"6e9863f26011b048cad576d856020723659a577fae0809fa877c08d06939c1a5"}
Oct 30 03:18:26.488671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:26.488631    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:26.578380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:26.578199    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-openshift-ca\") pod \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") "
Oct 30 03:18:26.578592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:26.578457    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-kube-api-access-vsd82\" (UniqueName: \"kubernetes.io/projected/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-kube-api-access-vsd82\") pod \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") "
Oct 30 03:18:26.578592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:26.578576    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-xks-ca\") pod \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\" (UID: \"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba\") "
Oct 30 03:18:26.585058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:26.584993    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-openshift-ca" pod "71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba" (UID: "71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:26.608897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:26.608844    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-kube-api-access-vsd82" pod "71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba" (UID: "71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba"). InnerVolumeSpecName "kube-api-access-vsd82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:26.623725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:26.623696    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-xks-ca" pod "71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba" (UID: "71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:26.680746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:26.680707    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsd82\" (UniqueName: \"kubernetes.io/projected/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-kube-api-access-vsd82\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:26.680746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:26.680738    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:26.680746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:26.680750    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:27.274020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:27.273988    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"
Oct 30 03:18:27.274207 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:27.274018    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm" event={"ID":"71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba","Type":"ContainerDied","Data":"42fc4d5cf148fbf801ed131183d4289cfd525922e6c43645a4296aa6e9870c70"}
Oct 30 03:18:27.274207 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:27.274076    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42fc4d5cf148fbf801ed131183d4289cfd525922e6c43645a4296aa6e9870c70"
Oct 30 03:18:29.534394 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.534356    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:18:29.537777 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.537745    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-credential-socket\") pod \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") "
Oct 30 03:18:29.537926 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.537799    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-socket\") pod \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") "
Oct 30 03:18:29.537926 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.537832    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-data\") pod \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") "
Oct 30 03:18:29.537926 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.537922    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-kube-api-access-8zg8p\" (UniqueName: \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-kube-api-access-8zg8p\") pod \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") "
Oct 30 03:18:29.538085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.537947    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-podinfo\") pod \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") "
Oct 30 03:18:29.538085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.538010    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-token\" (UniqueName: \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-token\") pod \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") "
Oct 30 03:18:29.538085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.538054    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-credential-socket" pod "dffbabd1-d7b2-4b61-aa48-58b0be40e9da" (UID: "dffbabd1-d7b2-4b61-aa48-58b0be40e9da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:29.538313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.538151    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-envoy\") pod \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") "
Oct 30 03:18:29.538313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.538195    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istiod-ca-cert\") pod \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") "
Oct 30 03:18:29.538313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.538220    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-socket" pod "dffbabd1-d7b2-4b61-aa48-58b0be40e9da" (UID: "dffbabd1-d7b2-4b61-aa48-58b0be40e9da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:29.538313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.538279    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-certs\") pod \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\" (UID: \"dffbabd1-d7b2-4b61-aa48-58b0be40e9da\") "
Oct 30 03:18:29.538743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.538716    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-data" pod "dffbabd1-d7b2-4b61-aa48-58b0be40e9da" (UID: "dffbabd1-d7b2-4b61-aa48-58b0be40e9da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:29.538820 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.538759    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istiod-ca-cert" pod "dffbabd1-d7b2-4b61-aa48-58b0be40e9da" (UID: "dffbabd1-d7b2-4b61-aa48-58b0be40e9da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:29.539204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.539168    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-certs" pod "dffbabd1-d7b2-4b61-aa48-58b0be40e9da" (UID: "dffbabd1-d7b2-4b61-aa48-58b0be40e9da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:29.541118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.541081    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.541270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.541124    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.541270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.541164    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.541270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.541180    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.541270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.541195    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.541701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.541672    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-envoy" pod "dffbabd1-d7b2-4b61-aa48-58b0be40e9da" (UID: "dffbabd1-d7b2-4b61-aa48-58b0be40e9da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:29.541822 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.541695    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-podinfo" pod "dffbabd1-d7b2-4b61-aa48-58b0be40e9da" (UID: "dffbabd1-d7b2-4b61-aa48-58b0be40e9da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:18:29.542159 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.542134    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-token" pod "dffbabd1-d7b2-4b61-aa48-58b0be40e9da" (UID: "dffbabd1-d7b2-4b61-aa48-58b0be40e9da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:29.542273 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.542169    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-kube-api-access-8zg8p" pod "dffbabd1-d7b2-4b61-aa48-58b0be40e9da" (UID: "dffbabd1-d7b2-4b61-aa48-58b0be40e9da"). InnerVolumeSpecName "kube-api-access-8zg8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:29.644673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.643838    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.644673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.643877    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zg8p\" (UniqueName: \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-kube-api-access-8zg8p\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.644673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.643897    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.644673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:29.643910    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffbabd1-d7b2-4b61-aa48-58b0be40e9da-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:30.304013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.303962    4278 generic.go:356] "Generic (PLEG): container finished" podID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerID="baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93" exitCode=0
Oct 30 03:18:30.304231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.304075    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" event={"ID":"dffbabd1-d7b2-4b61-aa48-58b0be40e9da","Type":"ContainerDied","Data":"baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93"}
Oct 30 03:18:30.304231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.304176    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7" event={"ID":"dffbabd1-d7b2-4b61-aa48-58b0be40e9da","Type":"ContainerDied","Data":"6e468d07e492f3284e41265fefb6ceb4e560ca00ab60071674c64e913b0b9c9f"}
Oct 30 03:18:30.304339 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.304275    4278 scope.go:122] "RemoveContainer" containerID="baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93"
Oct 30 03:18:30.304339 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.304314    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"
Oct 30 03:18:30.321471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.321449    4278 scope.go:122] "RemoveContainer" containerID="ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb"
Oct 30 03:18:30.331975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.331951    4278 scope.go:122] "RemoveContainer" containerID="baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93"
Oct 30 03:18:30.332330 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:18:30.332302    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93\": container with ID starting with baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93 not found: ID does not exist" containerID="baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93"
Oct 30 03:18:30.332380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.332345    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93"} err="failed to get container status \"baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93\": rpc error: code = NotFound desc = could not find container \"baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93\": container with ID starting with baba4af834167a918bb46d0660ebbe2da0ce36e180701183f9bb12c099cf6e93 not found: ID does not exist"
Oct 30 03:18:30.332380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.332372    4278 scope.go:122] "RemoveContainer" containerID="ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb"
Oct 30 03:18:30.332707 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:18:30.332686    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb\": container with ID starting with ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb not found: ID does not exist" containerID="ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb"
Oct 30 03:18:30.332769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.332712    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb"} err="failed to get container status \"ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb\": rpc error: code = NotFound desc = could not find container \"ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb\": container with ID starting with ad1235c62f40bfce5b77c25f120608b36a4970f8ba1c69ab16d75c84b46937bb not found: ID does not exist"
Oct 30 03:18:30.383061 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.383018    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"]
Oct 30 03:18:30.404497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:30.404451    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-2fvj7"]
Oct 30 03:18:31.650982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:31.650943    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"]
Oct 30 03:18:31.662759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:31.662717    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-16df10-gpgrb"]
Oct 30 03:18:32.203994 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.203955    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ae4668-6eb5-4d76-8416-b1ea5a354076" path="/var/lib/kubelet/pods/96ae4668-6eb5-4d76-8416-b1ea5a354076/volumes"
Oct 30 03:18:32.204787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.204766    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" path="/var/lib/kubelet/pods/dffbabd1-d7b2-4b61-aa48-58b0be40e9da/volumes"
Oct 30 03:18:32.254789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.254027    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"]
Oct 30 03:18:32.473596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.473480    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"]
Oct 30 03:18:32.474676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.474648    4278 cpu_manager.go:414] "containerMap: removing container" podUID="71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba" containerName="revoke-keys"
Oct 30 03:18:32.474676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.474676    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba" containerName="revoke-keys"
Oct 30 03:18:32.474852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.474694    4278 cpu_manager.go:414] "containerMap: removing container" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy"
Oct 30 03:18:32.474852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.474702    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy"
Oct 30 03:18:32.474852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.474727    4278 cpu_manager.go:414] "containerMap: removing container" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy"
Oct 30 03:18:32.474852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.474735    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy"
Oct 30 03:18:32.475082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.474952    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba" containerName="revoke-keys"
Oct 30 03:18:32.475082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.474968    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy"
Oct 30 03:18:32.475082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.474981    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffbabd1-d7b2-4b61-aa48-58b0be40e9da" containerName="istio-proxy"
Oct 30 03:18:32.480869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.480838    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.499414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.499385    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:18:32.505495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.505396    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:18:32.528448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.518601    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"]
Oct 30 03:18:32.606285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.606249    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4ee651-xj5s2\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.606542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.606520    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4ee651-xj5s2\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.606696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.606681    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvs8\" (UniqueName: \"kubernetes.io/projected/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-kube-api-access-5jvs8\") pod \"maas-api-revoke-sub-cf4ee651-xj5s2\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.708544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.708482    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4ee651-xj5s2\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.709052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.708711    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4ee651-xj5s2\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.709052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.708898    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvs8\" (UniqueName: \"kubernetes.io/projected/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-kube-api-access-5jvs8\") pod \"maas-api-revoke-sub-cf4ee651-xj5s2\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.709589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.709560    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4ee651-xj5s2\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.711149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.711124    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4ee651-xj5s2\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.729045 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.728982    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvs8\" (UniqueName: \"kubernetes.io/projected/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-kube-api-access-5jvs8\") pod \"maas-api-revoke-sub-cf4ee651-xj5s2\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.808264 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.808223    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:32.876311 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.876279    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"]
Oct 30 03:18:32.904961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.904522    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:32.904961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:32.904770    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"]
Oct 30 03:18:33.015412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.015364    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51eff22b-cgr7r\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:33.015701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.015677    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51eff22b-cgr7r\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:33.015768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.015752    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6ddr\" (UniqueName: \"kubernetes.io/projected/f23d2670-5f3b-4bb0-9635-795f367734dd-kube-api-access-w6ddr\") pod \"maas-api-revoke-sub-51eff22b-cgr7r\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:33.117835 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.117800    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51eff22b-cgr7r\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:33.117986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.117895    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ddr\" (UniqueName: \"kubernetes.io/projected/f23d2670-5f3b-4bb0-9635-795f367734dd-kube-api-access-w6ddr\") pod \"maas-api-revoke-sub-51eff22b-cgr7r\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:33.118119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.118085    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51eff22b-cgr7r\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:33.118710 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.118682    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"]
Oct 30 03:18:33.118843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.118822    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51eff22b-cgr7r\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:33.119326 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:33.119296    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5cfb36_3dfa_45d8_9df2_bac35fead58e.slice/crio-cd0c68b9d992a46b78864a112d1b38df52631051e88415f7ac3c5ce8ff3d5382 WatchSource:0}: Error finding container cd0c68b9d992a46b78864a112d1b38df52631051e88415f7ac3c5ce8ff3d5382: Status 404 returned error can't find the container with id cd0c68b9d992a46b78864a112d1b38df52631051e88415f7ac3c5ce8ff3d5382
Oct 30 03:18:33.120623 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.120601    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51eff22b-cgr7r\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:33.137944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.137913    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6ddr\" (UniqueName: \"kubernetes.io/projected/f23d2670-5f3b-4bb0-9635-795f367734dd-kube-api-access-w6ddr\") pod \"maas-api-revoke-sub-51eff22b-cgr7r\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:33.223711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.223671    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:33.344311 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.344144    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2" event={"ID":"0c5cfb36-3dfa-45d8-9df2-bac35fead58e","Type":"ContainerStarted","Data":"cd0c68b9d992a46b78864a112d1b38df52631051e88415f7ac3c5ce8ff3d5382"}
Oct 30 03:18:33.537163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:33.537088    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"]
Oct 30 03:18:33.537566 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:33.537533    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23d2670_5f3b_4bb0_9635_795f367734dd.slice/crio-c972faed496d29866db2124863fcfcb127fcf978c8502c6390b0bab969cfe516 WatchSource:0}: Error finding container c972faed496d29866db2124863fcfcb127fcf978c8502c6390b0bab969cfe516: Status 404 returned error can't find the container with id c972faed496d29866db2124863fcfcb127fcf978c8502c6390b0bab969cfe516
Oct 30 03:18:34.355414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:34.355371    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r" event={"ID":"f23d2670-5f3b-4bb0-9635-795f367734dd","Type":"ContainerStarted","Data":"f7dfb1ed6979157de8801927be9be2000b2f334f3893ac9e542574397f8b875a"}
Oct 30 03:18:34.355890 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:34.355453    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r" event={"ID":"f23d2670-5f3b-4bb0-9635-795f367734dd","Type":"ContainerStarted","Data":"c972faed496d29866db2124863fcfcb127fcf978c8502c6390b0bab969cfe516"}
Oct 30 03:18:34.357332 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:34.357298    4278 generic.go:356] "Generic (PLEG): container finished" podID="0c5cfb36-3dfa-45d8-9df2-bac35fead58e" containerID="a34998a7448928d6d06ed352c04a1e063a15e180ef9e40b2a52f0490539d5425" exitCode=0
Oct 30 03:18:34.357488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:34.357369    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2" event={"ID":"0c5cfb36-3dfa-45d8-9df2-bac35fead58e","Type":"ContainerDied","Data":"a34998a7448928d6d06ed352c04a1e063a15e180ef9e40b2a52f0490539d5425"}
Oct 30 03:18:34.398920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:34.398845    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r" podStartSLOduration=2.398825681 podStartE2EDuration="2.398825681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:34.395454872 +0000 UTC m=+2252.319663062" watchObservedRunningTime="2026-10-30 03:18:34.398825681 +0000 UTC m=+2252.323033869"
Oct 30 03:18:35.367760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.367724    4278 generic.go:356] "Generic (PLEG): container finished" podID="f23d2670-5f3b-4bb0-9635-795f367734dd" containerID="f7dfb1ed6979157de8801927be9be2000b2f334f3893ac9e542574397f8b875a" exitCode=0
Oct 30 03:18:35.368194 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.367799    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r" event={"ID":"f23d2670-5f3b-4bb0-9635-795f367734dd","Type":"ContainerDied","Data":"f7dfb1ed6979157de8801927be9be2000b2f334f3893ac9e542574397f8b875a"}
Oct 30 03:18:35.592933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.592872    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:35.770849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.770809    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-openshift-ca\") pod \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") "
Oct 30 03:18:35.770849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.770859    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-kube-api-access-5jvs8\" (UniqueName: \"kubernetes.io/projected/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-kube-api-access-5jvs8\") pod \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") "
Oct 30 03:18:35.771087 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.771057    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-xks-ca\") pod \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\" (UID: \"0c5cfb36-3dfa-45d8-9df2-bac35fead58e\") "
Oct 30 03:18:35.771290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.771260    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-openshift-ca" pod "0c5cfb36-3dfa-45d8-9df2-bac35fead58e" (UID: "0c5cfb36-3dfa-45d8-9df2-bac35fead58e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:35.771740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.771720    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:35.773778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.773750    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-xks-ca" pod "0c5cfb36-3dfa-45d8-9df2-bac35fead58e" (UID: "0c5cfb36-3dfa-45d8-9df2-bac35fead58e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:35.773891 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.773793    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-kube-api-access-5jvs8" pod "0c5cfb36-3dfa-45d8-9df2-bac35fead58e" (UID: "0c5cfb36-3dfa-45d8-9df2-bac35fead58e"). InnerVolumeSpecName "kube-api-access-5jvs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:35.873342 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.873298    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvs8\" (UniqueName: \"kubernetes.io/projected/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-kube-api-access-5jvs8\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:35.873342 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:35.873340    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5cfb36-3dfa-45d8-9df2-bac35fead58e-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:36.376908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.376869    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2" event={"ID":"0c5cfb36-3dfa-45d8-9df2-bac35fead58e","Type":"ContainerDied","Data":"cd0c68b9d992a46b78864a112d1b38df52631051e88415f7ac3c5ce8ff3d5382"}
Oct 30 03:18:36.376908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.376910    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"
Oct 30 03:18:36.377346 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.376928    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd0c68b9d992a46b78864a112d1b38df52631051e88415f7ac3c5ce8ff3d5382"
Oct 30 03:18:36.553875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.553847    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:36.685803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.685772    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23d2670-5f3b-4bb0-9635-795f367734dd-kube-api-access-w6ddr\" (UniqueName: \"kubernetes.io/projected/f23d2670-5f3b-4bb0-9635-795f367734dd-kube-api-access-w6ddr\") pod \"f23d2670-5f3b-4bb0-9635-795f367734dd\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") "
Oct 30 03:18:36.685967 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.685904    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-xks-ca\") pod \"f23d2670-5f3b-4bb0-9635-795f367734dd\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") "
Oct 30 03:18:36.686054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.686042    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-openshift-ca\") pod \"f23d2670-5f3b-4bb0-9635-795f367734dd\" (UID: \"f23d2670-5f3b-4bb0-9635-795f367734dd\") "
Oct 30 03:18:36.686445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.686397    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-openshift-ca" pod "f23d2670-5f3b-4bb0-9635-795f367734dd" (UID: "f23d2670-5f3b-4bb0-9635-795f367734dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:36.686725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.686698    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:36.688122 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.688090    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23d2670-5f3b-4bb0-9635-795f367734dd-kube-api-access-w6ddr" pod "f23d2670-5f3b-4bb0-9635-795f367734dd" (UID: "f23d2670-5f3b-4bb0-9635-795f367734dd"). InnerVolumeSpecName "kube-api-access-w6ddr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:36.688240 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.688119    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-xks-ca" pod "f23d2670-5f3b-4bb0-9635-795f367734dd" (UID: "f23d2670-5f3b-4bb0-9635-795f367734dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:36.788683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.788599    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23d2670-5f3b-4bb0-9635-795f367734dd-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:36.788683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:36.788629    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6ddr\" (UniqueName: \"kubernetes.io/projected/f23d2670-5f3b-4bb0-9635-795f367734dd-kube-api-access-w6ddr\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:37.387158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:37.387130    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"
Oct 30 03:18:37.387611 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:37.387122    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r" event={"ID":"f23d2670-5f3b-4bb0-9635-795f367734dd","Type":"ContainerDied","Data":"c972faed496d29866db2124863fcfcb127fcf978c8502c6390b0bab969cfe516"}
Oct 30 03:18:37.387611 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:37.387227    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c972faed496d29866db2124863fcfcb127fcf978c8502c6390b0bab969cfe516"
Oct 30 03:18:39.062967 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.062919    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"]
Oct 30 03:18:39.063659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.063641    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0c5cfb36-3dfa-45d8-9df2-bac35fead58e" containerName="revoke-keys"
Oct 30 03:18:39.063659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.063660    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5cfb36-3dfa-45d8-9df2-bac35fead58e" containerName="revoke-keys"
Oct 30 03:18:39.063808 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.063688    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f23d2670-5f3b-4bb0-9635-795f367734dd" containerName="revoke-keys"
Oct 30 03:18:39.063808 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.063694    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23d2670-5f3b-4bb0-9635-795f367734dd" containerName="revoke-keys"
Oct 30 03:18:39.063876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.063823    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5cfb36-3dfa-45d8-9df2-bac35fead58e" containerName="revoke-keys"
Oct 30 03:18:39.063876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.063837    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23d2670-5f3b-4bb0-9635-795f367734dd" containerName="revoke-keys"
Oct 30 03:18:39.067782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.067758    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.089387 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.089353    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"]
Oct 30 03:18:39.095734 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.095693    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:18:39.095876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.095770    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:18:39.123543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.123506    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-537001-zxrhz\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.123863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.123835    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-537001-zxrhz\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.123984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.123964    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2cc8\" (UniqueName: \"kubernetes.io/projected/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-kube-api-access-l2cc8\") pod \"maas-api-revoke-keys-e2e-inf-b-537001-zxrhz\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.225241 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.225199    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-537001-zxrhz\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.225462 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.225393    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-537001-zxrhz\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.225523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.225480    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2cc8\" (UniqueName: \"kubernetes.io/projected/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-kube-api-access-l2cc8\") pod \"maas-api-revoke-keys-e2e-inf-b-537001-zxrhz\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.226029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.226010    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-537001-zxrhz\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.227931 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.227913    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-537001-zxrhz\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.245765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.245736    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2cc8\" (UniqueName: \"kubernetes.io/projected/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-kube-api-access-l2cc8\") pod \"maas-api-revoke-keys-e2e-inf-b-537001-zxrhz\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.384417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.384338    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:39.693278 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:39.693227    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e64e43_b4bb_4e9f_ab96_6c11363f93dc.slice/crio-c5645eae981a548d8116e419366327ac5c120b010aeb88356cd6bfaae4bdcbb8 WatchSource:0}: Error finding container c5645eae981a548d8116e419366327ac5c120b010aeb88356cd6bfaae4bdcbb8: Status 404 returned error can't find the container with id c5645eae981a548d8116e419366327ac5c120b010aeb88356cd6bfaae4bdcbb8
Oct 30 03:18:39.693278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:39.693257    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"]
Oct 30 03:18:40.034825 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:40.034786    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"]
Oct 30 03:18:40.046914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:40.046876    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e553ae-858rr"]
Oct 30 03:18:40.205467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:40.205403    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c928965-d2d6-461f-953a-0ea65490b797" path="/var/lib/kubelet/pods/7c928965-d2d6-461f-953a-0ea65490b797/volumes"
Oct 30 03:18:40.413631 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:40.413587    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz" event={"ID":"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc","Type":"ContainerStarted","Data":"76cf74103bc08965cc9f201d37908e7bdff3f3b863fbe87eed443a56ed931623"}
Oct 30 03:18:40.413802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:40.413658    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz" event={"ID":"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc","Type":"ContainerStarted","Data":"c5645eae981a548d8116e419366327ac5c120b010aeb88356cd6bfaae4bdcbb8"}
Oct 30 03:18:40.462459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:40.462379    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz" podStartSLOduration=1.462360944 podStartE2EDuration="1.462360944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:40.461251564 +0000 UTC m=+2258.385459753" watchObservedRunningTime="2026-10-30 03:18:40.462360944 +0000 UTC m=+2258.386569134"
Oct 30 03:18:41.423455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:41.423374    4278 generic.go:356] "Generic (PLEG): container finished" podID="b7e64e43-b4bb-4e9f-ab96-6c11363f93dc" containerID="76cf74103bc08965cc9f201d37908e7bdff3f3b863fbe87eed443a56ed931623" exitCode=0
Oct 30 03:18:41.424043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:41.423519    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz" event={"ID":"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc","Type":"ContainerDied","Data":"76cf74103bc08965cc9f201d37908e7bdff3f3b863fbe87eed443a56ed931623"}
Oct 30 03:18:42.604633 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:42.604610    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:42.692643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:42.692550    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-openshift-ca\") pod \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") "
Oct 30 03:18:42.692643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:42.692617    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-kube-api-access-l2cc8\" (UniqueName: \"kubernetes.io/projected/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-kube-api-access-l2cc8\") pod \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") "
Oct 30 03:18:42.692896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:42.692800    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-xks-ca\") pod \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\" (UID: \"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc\") "
Oct 30 03:18:42.693021 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:42.692992    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-openshift-ca" pod "b7e64e43-b4bb-4e9f-ab96-6c11363f93dc" (UID: "b7e64e43-b4bb-4e9f-ab96-6c11363f93dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:42.693583 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:42.693558    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:42.695101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:42.695074    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-kube-api-access-l2cc8" pod "b7e64e43-b4bb-4e9f-ab96-6c11363f93dc" (UID: "b7e64e43-b4bb-4e9f-ab96-6c11363f93dc"). InnerVolumeSpecName "kube-api-access-l2cc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:42.695372 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:42.695354    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-xks-ca" pod "b7e64e43-b4bb-4e9f-ab96-6c11363f93dc" (UID: "b7e64e43-b4bb-4e9f-ab96-6c11363f93dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:42.795587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:42.795548    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:42.795587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:42.795588    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2cc8\" (UniqueName: \"kubernetes.io/projected/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc-kube-api-access-l2cc8\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:43.442063 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:43.442027    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz" event={"ID":"b7e64e43-b4bb-4e9f-ab96-6c11363f93dc","Type":"ContainerDied","Data":"c5645eae981a548d8116e419366327ac5c120b010aeb88356cd6bfaae4bdcbb8"}
Oct 30 03:18:43.442063 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:43.442054    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"
Oct 30 03:18:43.442305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:43.442077    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5645eae981a548d8116e419366327ac5c120b010aeb88356cd6bfaae4bdcbb8"
Oct 30 03:18:46.205899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:46.205864    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"]
Oct 30 03:18:47.286500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.286454    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r" podUID="b412a03c-6675-4269-a2e0-b0cd5a059d39" containerName="main" containerID="cri-o://88d31c31d1096dd781640f542b6cf1817c9eb4c18440d00b9bccaf6673ca7a8f" gracePeriod=45
Oct 30 03:18:47.503236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.502997    4278 generic.go:356] "Generic (PLEG): container finished" podID="b412a03c-6675-4269-a2e0-b0cd5a059d39" containerID="88d31c31d1096dd781640f542b6cf1817c9eb4c18440d00b9bccaf6673ca7a8f" exitCode=0
Oct 30 03:18:47.503236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.503179    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r" event={"ID":"b412a03c-6675-4269-a2e0-b0cd5a059d39","Type":"ContainerDied","Data":"88d31c31d1096dd781640f542b6cf1817c9eb4c18440d00b9bccaf6673ca7a8f"}
Oct 30 03:18:47.563299 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.563203    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"]
Oct 30 03:18:47.563521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.563489    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy" containerID="cri-o://b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce" gracePeriod=30
Oct 30 03:18:47.672919 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.672886    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:18:47.783519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.783484    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-model-cache\") pod \"b412a03c-6675-4269-a2e0-b0cd5a059d39\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") "
Oct 30 03:18:47.783702 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.783551    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-home\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-home\") pod \"b412a03c-6675-4269-a2e0-b0cd5a059d39\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") "
Oct 30 03:18:47.783702 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.783618    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b412a03c-6675-4269-a2e0-b0cd5a059d39-tls-certs\" (UniqueName: \"kubernetes.io/secret/b412a03c-6675-4269-a2e0-b0cd5a059d39-tls-certs\") pod \"b412a03c-6675-4269-a2e0-b0cd5a059d39\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") "
Oct 30 03:18:47.783796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.783768    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-model-cache" pod "b412a03c-6675-4269-a2e0-b0cd5a059d39" (UID: "b412a03c-6675-4269-a2e0-b0cd5a059d39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:47.783841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.783802    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-tmp-dir\") pod \"b412a03c-6675-4269-a2e0-b0cd5a059d39\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") "
Oct 30 03:18:47.783879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.783840    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b412a03c-6675-4269-a2e0-b0cd5a059d39-kube-api-access-dskst\" (UniqueName: \"kubernetes.io/projected/b412a03c-6675-4269-a2e0-b0cd5a059d39-kube-api-access-dskst\") pod \"b412a03c-6675-4269-a2e0-b0cd5a059d39\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") "
Oct 30 03:18:47.783879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.783860    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-home" pod "b412a03c-6675-4269-a2e0-b0cd5a059d39" (UID: "b412a03c-6675-4269-a2e0-b0cd5a059d39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:47.783961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.783895    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-dshm\") pod \"b412a03c-6675-4269-a2e0-b0cd5a059d39\" (UID: \"b412a03c-6675-4269-a2e0-b0cd5a059d39\") "
Oct 30 03:18:47.783961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.783924    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-tmp-dir" pod "b412a03c-6675-4269-a2e0-b0cd5a059d39" (UID: "b412a03c-6675-4269-a2e0-b0cd5a059d39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:47.784632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.784608    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.784632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.784629    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.784632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.784638    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.786029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.785995    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b412a03c-6675-4269-a2e0-b0cd5a059d39-tls-certs" pod "b412a03c-6675-4269-a2e0-b0cd5a059d39" (UID: "b412a03c-6675-4269-a2e0-b0cd5a059d39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:47.786117 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.786052    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-dshm" pod "b412a03c-6675-4269-a2e0-b0cd5a059d39" (UID: "b412a03c-6675-4269-a2e0-b0cd5a059d39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:47.786237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.786169    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b412a03c-6675-4269-a2e0-b0cd5a059d39-kube-api-access-dskst" pod "b412a03c-6675-4269-a2e0-b0cd5a059d39" (UID: "b412a03c-6675-4269-a2e0-b0cd5a059d39"). InnerVolumeSpecName "kube-api-access-dskst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:47.825035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.824961    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"]
Oct 30 03:18:47.825709 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.825688    4278 cpu_manager.go:414] "containerMap: removing container" podUID="b412a03c-6675-4269-a2e0-b0cd5a059d39" containerName="main"
Oct 30 03:18:47.825709 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.825708    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b412a03c-6675-4269-a2e0-b0cd5a059d39" containerName="main"
Oct 30 03:18:47.825856 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.825737    4278 cpu_manager.go:414] "containerMap: removing container" podUID="b7e64e43-b4bb-4e9f-ab96-6c11363f93dc" containerName="revoke-keys"
Oct 30 03:18:47.825856 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.825742    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7e64e43-b4bb-4e9f-ab96-6c11363f93dc" containerName="revoke-keys"
Oct 30 03:18:47.825922 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.825865    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7e64e43-b4bb-4e9f-ab96-6c11363f93dc" containerName="revoke-keys"
Oct 30 03:18:47.825922 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.825880    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="b412a03c-6675-4269-a2e0-b0cd5a059d39" containerName="main"
Oct 30 03:18:47.829694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.829674    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:47.858253 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.858221    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4cf8f5-openshift-default-dockercfg-zhgfj\""
Oct 30 03:18:47.867943 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.867870    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"]
Oct 30 03:18:47.886725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.886691    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b412a03c-6675-4269-a2e0-b0cd5a059d39-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.886876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.886737    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dskst\" (UniqueName: \"kubernetes.io/projected/b412a03c-6675-4269-a2e0-b0cd5a059d39-kube-api-access-dskst\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.886876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.886754    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b412a03c-6675-4269-a2e0-b0cd5a059d39-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.988321 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.988273    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-istio-token\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:47.988504 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.988331    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-credential-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:47.988504 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.988382    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2t9\" (UniqueName: \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-kube-api-access-8d2t9\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:47.988591 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.988512    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:47.988747 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.988712    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9443ad-9188-4827-ad53-609839b11f31-istiod-ca-cert\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:47.988870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.988851    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9443ad-9188-4827-ad53-609839b11f31-istio-podinfo\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:47.989011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.988995    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-data\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:47.989169 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.989154    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-envoy\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:47.989217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:47.989191    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-certs\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.110672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.110571    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9443ad-9188-4827-ad53-609839b11f31-istiod-ca-cert\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.111909 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.111884    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9443ad-9188-4827-ad53-609839b11f31-istio-podinfo\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.112196 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.112178    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-data\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.112440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.112409    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-envoy\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.116494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.112554    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-certs\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.116736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.116712    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-istio-token\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.116886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.116867    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-credential-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.120444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.116977    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2t9\" (UniqueName: \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-kube-api-access-8d2t9\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.120672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.120642    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.152446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.111779    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9443ad-9188-4827-ad53-609839b11f31-istiod-ca-cert\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.152446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.114193    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-data\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.152446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.120329    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-credential-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.152446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.120088    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-certs\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.152446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.120979    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.152446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.133480    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-envoy\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.152446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.137115    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9443ad-9188-4827-ad53-609839b11f31-istio-podinfo\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.178456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.178335    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-istio-token\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.182459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.182158    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2t9\" (UniqueName: \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-kube-api-access-8d2t9\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.447467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.447345    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:48.513727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.513696    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r" event={"ID":"b412a03c-6675-4269-a2e0-b0cd5a059d39","Type":"ContainerDied","Data":"46835f5bf52578bf0edee080a61832fe5263776fed25dd516a1f8152faf34690"}
Oct 30 03:18:48.513863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.513735    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"
Oct 30 03:18:48.513863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.513753    4278 scope.go:122] "RemoveContainer" containerID="88d31c31d1096dd781640f542b6cf1817c9eb4c18440d00b9bccaf6673ca7a8f"
Oct 30 03:18:48.584558 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.584519    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"]
Oct 30 03:18:48.599496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.591672    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f21058/test-model-f21058-kserve-6cbc669967-fm22r"]
Oct 30 03:18:48.774614 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.774577    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"]
Oct 30 03:18:48.775889 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:48.775854    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9443ad_9188_4827_ad53_609839b11f31.slice/crio-5f68f3adcd16051083d53700ffb7b0d57b5bc5ce425b92fd24408ee7afb3f1cb WatchSource:0}: Error finding container 5f68f3adcd16051083d53700ffb7b0d57b5bc5ce425b92fd24408ee7afb3f1cb: Status 404 returned error can't find the container with id 5f68f3adcd16051083d53700ffb7b0d57b5bc5ce425b92fd24408ee7afb3f1cb
Oct 30 03:18:48.779167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.779128    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:48.779313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.779223    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:48.779313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:48.779273    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:49.524418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:49.524375    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" event={"ID":"6b9443ad-9188-4827-ad53-609839b11f31","Type":"ContainerStarted","Data":"8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c"}
Oct 30 03:18:49.524914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:49.524456    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" event={"ID":"6b9443ad-9188-4827-ad53-609839b11f31","Type":"ContainerStarted","Data":"5f68f3adcd16051083d53700ffb7b0d57b5bc5ce425b92fd24408ee7afb3f1cb"}
Oct 30 03:18:49.585785 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:49.585707    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" podStartSLOduration=2.58568758 podStartE2EDuration="2.58568758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:49.584816896 +0000 UTC m=+2267.509025086" watchObservedRunningTime="2026-10-30 03:18:49.58568758 +0000 UTC m=+2267.509895769"
Oct 30 03:18:50.201636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:50.201596    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b412a03c-6675-4269-a2e0-b0cd5a059d39" path="/var/lib/kubelet/pods/b412a03c-6675-4269-a2e0-b0cd5a059d39/volumes"
Oct 30 03:18:50.456517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:50.447859    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:50.456517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:50.451826    4278 patch_prober.go:29] interesting pod/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:18:50.456517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:50.451885    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" podUID="6b9443ad-9188-4827-ad53-609839b11f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 03:18:51.029622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:51.029580    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"]
Oct 30 03:18:51.044707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:51.044668    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88627bdf-r674s"]
Oct 30 03:18:51.448265 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:51.448174    4278 patch_prober.go:29] interesting pod/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:18:51.448265 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:51.448234    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" podUID="6b9443ad-9188-4827-ad53-609839b11f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 03:18:52.203061 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:52.203026    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3e963a-e132-444b-8ff5-240ee0fa94b6" path="/var/lib/kubelet/pods/bb3e963a-e132-444b-8ff5-240ee0fa94b6/volumes"
Oct 30 03:18:52.447947 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:52.447911    4278 patch_prober.go:29] interesting pod/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:18:52.448135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:52.447967    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" podUID="6b9443ad-9188-4827-ad53-609839b11f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 03:18:52.998128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:52.998096    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:18:53.098338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.098282    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istiod-ca-cert\") pod \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") "
Oct 30 03:18:53.098563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.098493    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-data\") pod \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") "
Oct 30 03:18:53.098642 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.098564    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-certs\") pod \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") "
Oct 30 03:18:53.098642 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.098616    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-podinfo\") pod \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") "
Oct 30 03:18:53.098760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.098711    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-token\" (UniqueName: \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-token\") pod \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") "
Oct 30 03:18:53.098760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.098755    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-kube-api-access-lhr55\" (UniqueName: \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-kube-api-access-lhr55\") pod \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") "
Oct 30 03:18:53.098879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.098807    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-socket\") pod \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") "
Oct 30 03:18:53.098942 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.098915    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istiod-ca-cert" pod "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" (UID: "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:53.099005 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.098980    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-credential-socket\") pod \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") "
Oct 30 03:18:53.099070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.099022    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-envoy\") pod \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\" (UID: \"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f\") "
Oct 30 03:18:53.099544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.099209    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-certs" pod "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" (UID: "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:53.099544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.099229    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-data" pod "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" (UID: "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:53.100086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.100058    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:53.100208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.100089    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:53.100208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.100103    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:53.100439 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.100382    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-credential-socket" pod "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" (UID: "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:53.100691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.100626    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-socket" pod "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" (UID: "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:53.102338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.102314    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-kube-api-access-lhr55" pod "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" (UID: "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f"). InnerVolumeSpecName "kube-api-access-lhr55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:53.102683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.102645    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-token" pod "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" (UID: "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:53.104269 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.104242    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-envoy" pod "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" (UID: "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:53.108300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.108258    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-podinfo" pod "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" (UID: "2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:18:53.201865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.201765    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:53.201865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.201798    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:53.201865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.201811    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhr55\" (UniqueName: \"kubernetes.io/projected/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-kube-api-access-lhr55\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:53.201865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.201824    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:53.201865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.201836    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:53.201865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.201847    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:18:53.447886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.447844    4278 patch_prober.go:29] interesting pod/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:18:53.448300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.447904    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" podUID="6b9443ad-9188-4827-ad53-609839b11f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 03:18:53.589536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.588898    4278 generic.go:356] "Generic (PLEG): container finished" podID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerID="b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce" exitCode=0
Oct 30 03:18:53.589536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.588970    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"
Oct 30 03:18:53.589536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.589209    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" event={"ID":"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f","Type":"ContainerDied","Data":"b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce"}
Oct 30 03:18:53.589536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.589273    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4" event={"ID":"2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f","Type":"ContainerDied","Data":"d8a813e2443b10f77ddf4e5d12a888c3a3fd89d0da66e21dac37fae1d1bf9d4e"}
Oct 30 03:18:53.589536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.589321    4278 scope.go:122] "RemoveContainer" containerID="b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce"
Oct 30 03:18:53.603712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.603690    4278 scope.go:122] "RemoveContainer" containerID="b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce"
Oct 30 03:18:53.603997 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:18:53.603971    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce\": container with ID starting with b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce not found: ID does not exist" containerID="b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce"
Oct 30 03:18:53.604095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.604004    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce"} err="failed to get container status \"b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce\": rpc error: code = NotFound desc = could not find container \"b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce\": container with ID starting with b17ef3e190f567d6d0cc7b4dd26454c32f927e4f5fc4bb6a01f88c8d1e6da9ce not found: ID does not exist"
Oct 30 03:18:53.646273 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.646242    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"]
Oct 30 03:18:53.658462 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:53.658395    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mkrp4"]
Oct 30 03:18:54.204188 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:54.204151    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" path="/var/lib/kubelet/pods/2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f/volumes"
Oct 30 03:18:54.213825 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:18:54.213789    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:18:54.448400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:54.448363    4278 patch_prober.go:29] interesting pod/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:18:54.448846 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:54.448441    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" podUID="6b9443ad-9188-4827-ad53-609839b11f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 03:18:55.483520 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:55.483415    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:55.484049 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:55.483863    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:55.484873 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:55.484839    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:18:58.606622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.606583    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"]
Oct 30 03:18:58.839511 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.837201    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"]
Oct 30 03:18:58.839511 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.838399    4278 cpu_manager.go:414] "containerMap: removing container" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy"
Oct 30 03:18:58.839511 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.838617    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy"
Oct 30 03:18:58.839511 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.838959    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1061dc-fc7f-48e3-89ce-e1a691ab0b7f" containerName="istio-proxy"
Oct 30 03:18:58.843449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.843413    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:58.853694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.853658    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"]
Oct 30 03:18:58.860903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.860833    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:18:58.861059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.860906    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:18:58.947041 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.946992    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f21058-9z26j\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:58.947257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.947066    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f21058-9z26j\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:58.947417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:58.947393    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr8kn\" (UniqueName: \"kubernetes.io/projected/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-kube-api-access-zr8kn\") pod \"maas-api-revoke-keys-e2e-inf-a-f21058-9z26j\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:59.049217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:59.049177    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr8kn\" (UniqueName: \"kubernetes.io/projected/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-kube-api-access-zr8kn\") pod \"maas-api-revoke-keys-e2e-inf-a-f21058-9z26j\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:59.049486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:59.049286    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f21058-9z26j\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:59.049486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:59.049336    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f21058-9z26j\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:59.050225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:59.050194    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f21058-9z26j\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:59.052274 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:59.052243    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f21058-9z26j\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:59.069232 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:59.069200    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr8kn\" (UniqueName: \"kubernetes.io/projected/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-kube-api-access-zr8kn\") pod \"maas-api-revoke-keys-e2e-inf-a-f21058-9z26j\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:59.168236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:59.168149    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:18:59.483009 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:59.482971    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"]
Oct 30 03:18:59.483550 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:18:59.483521    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d1dd7f_33eb_4cad_ac8d_e82949faeb0d.slice/crio-aa54e8f90725601e7337ce596184508d839cbdcdda3d5b0ca1caf7399439b8fb WatchSource:0}: Error finding container aa54e8f90725601e7337ce596184508d839cbdcdda3d5b0ca1caf7399439b8fb: Status 404 returned error can't find the container with id aa54e8f90725601e7337ce596184508d839cbdcdda3d5b0ca1caf7399439b8fb
Oct 30 03:18:59.660230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:18:59.660188    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j" event={"ID":"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d","Type":"ContainerStarted","Data":"aa54e8f90725601e7337ce596184508d839cbdcdda3d5b0ca1caf7399439b8fb"}
Oct 30 03:19:00.671131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:00.671032    4278 generic.go:356] "Generic (PLEG): container finished" podID="c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d" containerID="4307b81ed37caa4b3fe9e8ecfaa72e775d2b71ccfa1ab2c9a556a03fd5ce4574" exitCode=0
Oct 30 03:19:00.671559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:00.671125    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j" event={"ID":"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d","Type":"ContainerDied","Data":"4307b81ed37caa4b3fe9e8ecfaa72e775d2b71ccfa1ab2c9a556a03fd5ce4574"}
Oct 30 03:19:01.228594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.228548    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk" podUID="4e362931-63ca-4a40-8c39-2725e38cdc3d" containerName="main" containerID="cri-o://25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135" gracePeriod=45
Oct 30 03:19:01.231665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.231630    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk" podUID="4e362931-63ca-4a40-8c39-2725e38cdc3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.161:8000/ready\": write tcp 10.134.0.2:40154->10.134.0.161:8000: write: connection reset by peer"
Oct 30 03:19:01.544038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.544002    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:19:01.591580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.591533    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-model-cache\") pod \"4e362931-63ca-4a40-8c39-2725e38cdc3d\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") "
Oct 30 03:19:01.591791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.591613    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e362931-63ca-4a40-8c39-2725e38cdc3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e362931-63ca-4a40-8c39-2725e38cdc3d-tls-certs\") pod \"4e362931-63ca-4a40-8c39-2725e38cdc3d\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") "
Oct 30 03:19:01.591855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.591812    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e362931-63ca-4a40-8c39-2725e38cdc3d-kube-api-access-tpsls\" (UniqueName: \"kubernetes.io/projected/4e362931-63ca-4a40-8c39-2725e38cdc3d-kube-api-access-tpsls\") pod \"4e362931-63ca-4a40-8c39-2725e38cdc3d\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") "
Oct 30 03:19:01.591855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.591828    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-model-cache" pod "4e362931-63ca-4a40-8c39-2725e38cdc3d" (UID: "4e362931-63ca-4a40-8c39-2725e38cdc3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:01.591977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.591878    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-dshm\") pod \"4e362931-63ca-4a40-8c39-2725e38cdc3d\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") "
Oct 30 03:19:01.591977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.591902    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-home\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-home\") pod \"4e362931-63ca-4a40-8c39-2725e38cdc3d\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") "
Oct 30 03:19:01.592112 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.592092    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-tmp-dir\") pod \"4e362931-63ca-4a40-8c39-2725e38cdc3d\" (UID: \"4e362931-63ca-4a40-8c39-2725e38cdc3d\") "
Oct 30 03:19:01.592520 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.592484    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-home" pod "4e362931-63ca-4a40-8c39-2725e38cdc3d" (UID: "4e362931-63ca-4a40-8c39-2725e38cdc3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:01.592741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.592706    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-tmp-dir" pod "4e362931-63ca-4a40-8c39-2725e38cdc3d" (UID: "4e362931-63ca-4a40-8c39-2725e38cdc3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:01.592973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.592954    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:01.593035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.592978    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:01.593035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.592992    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:01.594556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.594527    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e362931-63ca-4a40-8c39-2725e38cdc3d-kube-api-access-tpsls" pod "4e362931-63ca-4a40-8c39-2725e38cdc3d" (UID: "4e362931-63ca-4a40-8c39-2725e38cdc3d"). InnerVolumeSpecName "kube-api-access-tpsls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:01.595756 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.595732    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e362931-63ca-4a40-8c39-2725e38cdc3d-tls-certs" pod "4e362931-63ca-4a40-8c39-2725e38cdc3d" (UID: "4e362931-63ca-4a40-8c39-2725e38cdc3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:01.596004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.595989    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-dshm" pod "4e362931-63ca-4a40-8c39-2725e38cdc3d" (UID: "4e362931-63ca-4a40-8c39-2725e38cdc3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:01.682464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.682379    4278 generic.go:356] "Generic (PLEG): container finished" podID="4e362931-63ca-4a40-8c39-2725e38cdc3d" containerID="25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135" exitCode=0
Oct 30 03:19:01.682965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.682479    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk" event={"ID":"4e362931-63ca-4a40-8c39-2725e38cdc3d","Type":"ContainerDied","Data":"25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135"}
Oct 30 03:19:01.682965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.682540    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk" event={"ID":"4e362931-63ca-4a40-8c39-2725e38cdc3d","Type":"ContainerDied","Data":"ce8f1a650aa7bb70aefb243d084f492406cb21661134a014a1c6ca2cd395a795"}
Oct 30 03:19:01.682965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.682542    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"
Oct 30 03:19:01.682965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.682557    4278 scope.go:122] "RemoveContainer" containerID="25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135"
Oct 30 03:19:01.695741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.695369    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e362931-63ca-4a40-8c39-2725e38cdc3d-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:01.695741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.695410    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpsls\" (UniqueName: \"kubernetes.io/projected/4e362931-63ca-4a40-8c39-2725e38cdc3d-kube-api-access-tpsls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:01.695741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.695445    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e362931-63ca-4a40-8c39-2725e38cdc3d-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:01.699508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.699473    4278 scope.go:122] "RemoveContainer" containerID="25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135"
Oct 30 03:19:01.699914 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:19:01.699891    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135\": container with ID starting with 25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135 not found: ID does not exist" containerID="25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135"
Oct 30 03:19:01.700028 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.699925    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135"} err="failed to get container status \"25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135\": rpc error: code = NotFound desc = could not find container \"25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135\": container with ID starting with 25c86f4a21359a614a37cee01eff0332f76c1c75fc45e95e784951d9f4050135 not found: ID does not exist"
Oct 30 03:19:01.842335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.842292    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"]
Oct 30 03:19:01.843392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.843370    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/sub-test-model-9245d1-kserve-655d677855-9kwmk"]
Oct 30 03:19:01.921635 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.919030    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"]
Oct 30 03:19:01.921635 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.920181    4278 cpu_manager.go:414] "containerMap: removing container" podUID="4e362931-63ca-4a40-8c39-2725e38cdc3d" containerName="main"
Oct 30 03:19:01.921635 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.920206    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e362931-63ca-4a40-8c39-2725e38cdc3d" containerName="main"
Oct 30 03:19:01.921635 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.920459    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e362931-63ca-4a40-8c39-2725e38cdc3d" containerName="main"
Oct 30 03:19:01.928345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.928298    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:01.951622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:01.951571    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"]
Oct 30 03:19:02.008872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.008847    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:19:02.010571 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.010546    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:02.010734 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.010711    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8cz\" (UniqueName: \"kubernetes.io/projected/30cab217-62f5-4c19-a400-139bff6eb512-kube-api-access-qh8cz\") pod \"maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:02.010823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.010805    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:02.112335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.112239    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-kube-api-access-zr8kn\" (UniqueName: \"kubernetes.io/projected/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-kube-api-access-zr8kn\") pod \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") "
Oct 30 03:19:02.112562 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.112527    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-openshift-ca\") pod \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") "
Oct 30 03:19:02.112714 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.112593    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-xks-ca\") pod \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\" (UID: \"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d\") "
Oct 30 03:19:02.112946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.112921    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-openshift-ca" pod "c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d" (UID: "c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:02.113031 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.113013    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:02.113189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.113170    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8cz\" (UniqueName: \"kubernetes.io/projected/30cab217-62f5-4c19-a400-139bff6eb512-kube-api-access-qh8cz\") pod \"maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:02.113286 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.113264    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:02.113695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.113655    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:02.114062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.114036    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:02.115022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.114992    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-kube-api-access-zr8kn" pod "c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d" (UID: "c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d"). InnerVolumeSpecName "kube-api-access-zr8kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:02.115131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.115121    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-xks-ca" pod "c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d" (UID: "c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:02.115731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.115701    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:02.136195 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.136155    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8cz\" (UniqueName: \"kubernetes.io/projected/30cab217-62f5-4c19-a400-139bff6eb512-kube-api-access-qh8cz\") pod \"maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:02.201949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.201916    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e362931-63ca-4a40-8c39-2725e38cdc3d" path="/var/lib/kubelet/pods/4e362931-63ca-4a40-8c39-2725e38cdc3d/volumes"
Oct 30 03:19:02.215687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.215641    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr8kn\" (UniqueName: \"kubernetes.io/projected/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-kube-api-access-zr8kn\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:02.215687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.215686    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:02.256824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.256793    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:02.570872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.570832    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"]
Oct 30 03:19:02.572176 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:19:02.572140    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cab217_62f5_4c19_a400_139bff6eb512.slice/crio-4a835d9a657b09d443c6ae421bcf7de4ee9b3cc2cbd5afec2a9d325dd21a23e6 WatchSource:0}: Error finding container 4a835d9a657b09d443c6ae421bcf7de4ee9b3cc2cbd5afec2a9d325dd21a23e6: Status 404 returned error can't find the container with id 4a835d9a657b09d443c6ae421bcf7de4ee9b3cc2cbd5afec2a9d325dd21a23e6
Oct 30 03:19:02.692847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.692808    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j" event={"ID":"c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d","Type":"ContainerDied","Data":"aa54e8f90725601e7337ce596184508d839cbdcdda3d5b0ca1caf7399439b8fb"}
Oct 30 03:19:02.693359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.692895    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa54e8f90725601e7337ce596184508d839cbdcdda3d5b0ca1caf7399439b8fb"
Oct 30 03:19:02.693359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.692820    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"
Oct 30 03:19:02.694262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:02.694238    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56" event={"ID":"30cab217-62f5-4c19-a400-139bff6eb512","Type":"ContainerStarted","Data":"4a835d9a657b09d443c6ae421bcf7de4ee9b3cc2cbd5afec2a9d325dd21a23e6"}
Oct 30 03:19:03.704371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:03.704331    4278 generic.go:356] "Generic (PLEG): container finished" podID="30cab217-62f5-4c19-a400-139bff6eb512" containerID="9d65d3d66c370d56e37bf2e93fcadddec9c0750b29a2288e76642239e36792e8" exitCode=0
Oct 30 03:19:03.704817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:03.704436    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56" event={"ID":"30cab217-62f5-4c19-a400-139bff6eb512","Type":"ContainerDied","Data":"9d65d3d66c370d56e37bf2e93fcadddec9c0750b29a2288e76642239e36792e8"}
Oct 30 03:19:04.122972 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:04.122936    4278 scope.go:122] "RemoveContainer" containerID="d038ccb4948a057769a3ebf2f8fdd013222d0d26cca03f6d5ed9ac4ff5eb4edb"
Oct 30 03:19:04.133786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:04.133755    4278 scope.go:122] "RemoveContainer" containerID="ae6db0e6ed003f3319408ea89c081442da4c376dc82c5fed5d7dc1f12fdd8297"
Oct 30 03:19:04.888858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:04.888829    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:04.958343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:04.958311    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-xks-ca\") pod \"30cab217-62f5-4c19-a400-139bff6eb512\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") "
Oct 30 03:19:04.958704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:04.958680    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30cab217-62f5-4c19-a400-139bff6eb512-kube-api-access-qh8cz\" (UniqueName: \"kubernetes.io/projected/30cab217-62f5-4c19-a400-139bff6eb512-kube-api-access-qh8cz\") pod \"30cab217-62f5-4c19-a400-139bff6eb512\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") "
Oct 30 03:19:04.958791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:04.958775    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-openshift-ca\") pod \"30cab217-62f5-4c19-a400-139bff6eb512\" (UID: \"30cab217-62f5-4c19-a400-139bff6eb512\") "
Oct 30 03:19:04.959212 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:04.959182    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-openshift-ca" pod "30cab217-62f5-4c19-a400-139bff6eb512" (UID: "30cab217-62f5-4c19-a400-139bff6eb512"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:04.959534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:04.959514    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:04.961080 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:04.961047    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cab217-62f5-4c19-a400-139bff6eb512-kube-api-access-qh8cz" pod "30cab217-62f5-4c19-a400-139bff6eb512" (UID: "30cab217-62f5-4c19-a400-139bff6eb512"). InnerVolumeSpecName "kube-api-access-qh8cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:04.961210 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:04.961109    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-xks-ca" pod "30cab217-62f5-4c19-a400-139bff6eb512" (UID: "30cab217-62f5-4c19-a400-139bff6eb512"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:05.061541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:05.061500    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh8cz\" (UniqueName: \"kubernetes.io/projected/30cab217-62f5-4c19-a400-139bff6eb512-kube-api-access-qh8cz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:05.061541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:05.061537    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cab217-62f5-4c19-a400-139bff6eb512-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:05.725675 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:05.725647    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"
Oct 30 03:19:05.725877 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:05.725676    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56" event={"ID":"30cab217-62f5-4c19-a400-139bff6eb512","Type":"ContainerDied","Data":"4a835d9a657b09d443c6ae421bcf7de4ee9b3cc2cbd5afec2a9d325dd21a23e6"}
Oct 30 03:19:05.725877 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:05.725729    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a835d9a657b09d443c6ae421bcf7de4ee9b3cc2cbd5afec2a9d325dd21a23e6"
Oct 30 03:19:13.631703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:13.631660    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk" podUID="18642810-698e-4f33-a220-dd28c6b54949" containerName="main" containerID="cri-o://a8b76446f7f82034488543d78ac173f33dd93ad55e748dc07903fdd34bd6f95d" gracePeriod=45
Oct 30 03:19:13.797866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:13.797831    4278 generic.go:356] "Generic (PLEG): container finished" podID="18642810-698e-4f33-a220-dd28c6b54949" containerID="a8b76446f7f82034488543d78ac173f33dd93ad55e748dc07903fdd34bd6f95d" exitCode=0
Oct 30 03:19:13.798037 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:13.797908    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk" event={"ID":"18642810-698e-4f33-a220-dd28c6b54949","Type":"ContainerDied","Data":"a8b76446f7f82034488543d78ac173f33dd93ad55e748dc07903fdd34bd6f95d"}
Oct 30 03:19:13.915801 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:13.915776    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:19:14.019386 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.019349    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-model-cache\") pod \"18642810-698e-4f33-a220-dd28c6b54949\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") "
Oct 30 03:19:14.019637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.019439    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-tmp-dir\") pod \"18642810-698e-4f33-a220-dd28c6b54949\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") "
Oct 30 03:19:14.019637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.019461    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-home\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-home\") pod \"18642810-698e-4f33-a220-dd28c6b54949\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") "
Oct 30 03:19:14.019637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.019490    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-dshm\") pod \"18642810-698e-4f33-a220-dd28c6b54949\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") "
Oct 30 03:19:14.019637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.019566    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18642810-698e-4f33-a220-dd28c6b54949-kube-api-access-vg9h5\" (UniqueName: \"kubernetes.io/projected/18642810-698e-4f33-a220-dd28c6b54949-kube-api-access-vg9h5\") pod \"18642810-698e-4f33-a220-dd28c6b54949\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") "
Oct 30 03:19:14.019858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.019732    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-model-cache" pod "18642810-698e-4f33-a220-dd28c6b54949" (UID: "18642810-698e-4f33-a220-dd28c6b54949"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:14.019858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.019747    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-tmp-dir" pod "18642810-698e-4f33-a220-dd28c6b54949" (UID: "18642810-698e-4f33-a220-dd28c6b54949"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:14.019858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.019759    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-home" pod "18642810-698e-4f33-a220-dd28c6b54949" (UID: "18642810-698e-4f33-a220-dd28c6b54949"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:14.019858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.019773    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18642810-698e-4f33-a220-dd28c6b54949-tls-certs\" (UniqueName: \"kubernetes.io/secret/18642810-698e-4f33-a220-dd28c6b54949-tls-certs\") pod \"18642810-698e-4f33-a220-dd28c6b54949\" (UID: \"18642810-698e-4f33-a220-dd28c6b54949\") "
Oct 30 03:19:14.020295 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.020273    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:14.020295 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.020291    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:14.020295 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.020298    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:14.021790 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.021761    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-dshm" pod "18642810-698e-4f33-a220-dd28c6b54949" (UID: "18642810-698e-4f33-a220-dd28c6b54949"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:14.021910 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.021872    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18642810-698e-4f33-a220-dd28c6b54949-kube-api-access-vg9h5" pod "18642810-698e-4f33-a220-dd28c6b54949" (UID: "18642810-698e-4f33-a220-dd28c6b54949"). InnerVolumeSpecName "kube-api-access-vg9h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:14.021958 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.021912    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18642810-698e-4f33-a220-dd28c6b54949-tls-certs" pod "18642810-698e-4f33-a220-dd28c6b54949" (UID: "18642810-698e-4f33-a220-dd28c6b54949"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:14.121839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.121804    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18642810-698e-4f33-a220-dd28c6b54949-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:14.121839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.121837    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg9h5\" (UniqueName: \"kubernetes.io/projected/18642810-698e-4f33-a220-dd28c6b54949-kube-api-access-vg9h5\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:14.122055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.121851    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18642810-698e-4f33-a220-dd28c6b54949-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:14.807345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.807259    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"
Oct 30 03:19:14.807345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.807290    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk" event={"ID":"18642810-698e-4f33-a220-dd28c6b54949","Type":"ContainerDied","Data":"50c0116065ae44d92cd422038161f4a7cea67809427f17f074b6ce9311440bf8"}
Oct 30 03:19:14.807887 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.807348    4278 scope.go:122] "RemoveContainer" containerID="a8b76446f7f82034488543d78ac173f33dd93ad55e748dc07903fdd34bd6f95d"
Oct 30 03:19:14.863075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.863039    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"]
Oct 30 03:19:14.875279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:14.875253    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2305f4/test-model-2305f4-kserve-574fd5b8b7-rvxxk"]
Oct 30 03:19:16.452452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:16.450641    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18642810-698e-4f33-a220-dd28c6b54949" path="/var/lib/kubelet/pods/18642810-698e-4f33-a220-dd28c6b54949/volumes"
Oct 30 03:19:17.009914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:17.009874    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"]
Oct 30 03:19:17.017368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:17.017334    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-537001-zxrhz"]
Oct 30 03:19:18.202784 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:18.202752    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7e64e43-b4bb-4e9f-ab96-6c11363f93dc" path="/var/lib/kubelet/pods/b7e64e43-b4bb-4e9f-ab96-6c11363f93dc/volumes"
Oct 30 03:19:22.718188 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:22.718147    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"]
Oct 30 03:19:22.730797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:22.730763    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7f767-sqj56"]
Oct 30 03:19:24.192645 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:19:24.192599    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:19:24.201585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.201551    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30cab217-62f5-4c19-a400-139bff6eb512" path="/var/lib/kubelet/pods/30cab217-62f5-4c19-a400-139bff6eb512/volumes"
Oct 30 03:19:24.913473 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.913417    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"]
Oct 30 03:19:24.914167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.914145    4278 cpu_manager.go:414] "containerMap: removing container" podUID="30cab217-62f5-4c19-a400-139bff6eb512" containerName="revoke-keys"
Oct 30 03:19:24.914279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.914168    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30cab217-62f5-4c19-a400-139bff6eb512" containerName="revoke-keys"
Oct 30 03:19:24.914279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.914186    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d" containerName="revoke-keys"
Oct 30 03:19:24.914279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.914193    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d" containerName="revoke-keys"
Oct 30 03:19:24.914279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.914218    4278 cpu_manager.go:414] "containerMap: removing container" podUID="18642810-698e-4f33-a220-dd28c6b54949" containerName="main"
Oct 30 03:19:24.914279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.914228    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18642810-698e-4f33-a220-dd28c6b54949" containerName="main"
Oct 30 03:19:24.914579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.914440    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="30cab217-62f5-4c19-a400-139bff6eb512" containerName="revoke-keys"
Oct 30 03:19:24.914579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.914457    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d" containerName="revoke-keys"
Oct 30 03:19:24.914579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.914470    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="18642810-698e-4f33-a220-dd28c6b54949" containerName="main"
Oct 30 03:19:24.919930 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.919908    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:24.940017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.939984    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"]
Oct 30 03:19:24.946925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.946898    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:19:24.947079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:24.946953    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:19:25.012910 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.012872    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682dd073-dtc6f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:25.013121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.012996    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682dd073-dtc6f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:25.013121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.013028    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl7hk\" (UniqueName: \"kubernetes.io/projected/4ba7d411-d429-4dd7-b9f9-517d228c399f-kube-api-access-zl7hk\") pod \"maas-api-revoke-sub-682dd073-dtc6f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:25.125892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.125827    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682dd073-dtc6f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:25.126179 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.126161    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682dd073-dtc6f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:25.126228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.126194    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl7hk\" (UniqueName: \"kubernetes.io/projected/4ba7d411-d429-4dd7-b9f9-517d228c399f-kube-api-access-zl7hk\") pod \"maas-api-revoke-sub-682dd073-dtc6f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:25.126940 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.126916    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682dd073-dtc6f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:25.128728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.128705    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682dd073-dtc6f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:25.146412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.146378    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl7hk\" (UniqueName: \"kubernetes.io/projected/4ba7d411-d429-4dd7-b9f9-517d228c399f-kube-api-access-zl7hk\") pod \"maas-api-revoke-sub-682dd073-dtc6f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:25.234793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.234715    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:25.544962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.544919    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"]
Oct 30 03:19:25.546903 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:19:25.546877    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba7d411_d429_4dd7_b9f9_517d228c399f.slice/crio-d253cd74d2ae8043e2876a400193d540d14a84437ece41312f746dbd8c761312 WatchSource:0}: Error finding container d253cd74d2ae8043e2876a400193d540d14a84437ece41312f746dbd8c761312: Status 404 returned error can't find the container with id d253cd74d2ae8043e2876a400193d540d14a84437ece41312f746dbd8c761312
Oct 30 03:19:25.946678 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:25.946508    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f" event={"ID":"4ba7d411-d429-4dd7-b9f9-517d228c399f","Type":"ContainerStarted","Data":"d253cd74d2ae8043e2876a400193d540d14a84437ece41312f746dbd8c761312"}
Oct 30 03:19:26.957729 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:26.957612    4278 generic.go:356] "Generic (PLEG): container finished" podID="4ba7d411-d429-4dd7-b9f9-517d228c399f" containerID="ad126baef012f2f0c2b7354e9cf5c30a697457c30d1f25b5cf7d74497debbe9d" exitCode=0
Oct 30 03:19:26.958223 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:26.957766    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f" event={"ID":"4ba7d411-d429-4dd7-b9f9-517d228c399f","Type":"ContainerDied","Data":"ad126baef012f2f0c2b7354e9cf5c30a697457c30d1f25b5cf7d74497debbe9d"}
Oct 30 03:19:28.153990 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.153961    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:28.195446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.195387    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba7d411-d429-4dd7-b9f9-517d228c399f-kube-api-access-zl7hk\" (UniqueName: \"kubernetes.io/projected/4ba7d411-d429-4dd7-b9f9-517d228c399f-kube-api-access-zl7hk\") pod \"4ba7d411-d429-4dd7-b9f9-517d228c399f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") "
Oct 30 03:19:28.195698 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.195668    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-xks-ca\") pod \"4ba7d411-d429-4dd7-b9f9-517d228c399f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") "
Oct 30 03:19:28.196630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.195857    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-openshift-ca\") pod \"4ba7d411-d429-4dd7-b9f9-517d228c399f\" (UID: \"4ba7d411-d429-4dd7-b9f9-517d228c399f\") "
Oct 30 03:19:28.197178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.197148    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-openshift-ca" pod "4ba7d411-d429-4dd7-b9f9-517d228c399f" (UID: "4ba7d411-d429-4dd7-b9f9-517d228c399f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:28.199999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.199946    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:28.204710 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.204262    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-xks-ca" pod "4ba7d411-d429-4dd7-b9f9-517d228c399f" (UID: "4ba7d411-d429-4dd7-b9f9-517d228c399f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:28.207811 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.207750    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba7d411-d429-4dd7-b9f9-517d228c399f-kube-api-access-zl7hk" pod "4ba7d411-d429-4dd7-b9f9-517d228c399f" (UID: "4ba7d411-d429-4dd7-b9f9-517d228c399f"). InnerVolumeSpecName "kube-api-access-zl7hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:28.301700 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.301666    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl7hk\" (UniqueName: \"kubernetes.io/projected/4ba7d411-d429-4dd7-b9f9-517d228c399f-kube-api-access-zl7hk\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:28.301700 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.301696    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba7d411-d429-4dd7-b9f9-517d228c399f-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:28.984280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.984237    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f" event={"ID":"4ba7d411-d429-4dd7-b9f9-517d228c399f","Type":"ContainerDied","Data":"d253cd74d2ae8043e2876a400193d540d14a84437ece41312f746dbd8c761312"}
Oct 30 03:19:28.984536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.984304    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d253cd74d2ae8043e2876a400193d540d14a84437ece41312f746dbd8c761312"
Oct 30 03:19:28.984536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:28.984364    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"
Oct 30 03:19:30.750282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.750243    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"]
Oct 30 03:19:30.751256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.751224    4278 cpu_manager.go:414] "containerMap: removing container" podUID="4ba7d411-d429-4dd7-b9f9-517d228c399f" containerName="revoke-keys"
Oct 30 03:19:30.751256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.751254    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba7d411-d429-4dd7-b9f9-517d228c399f" containerName="revoke-keys"
Oct 30 03:19:30.751566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.751505    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba7d411-d429-4dd7-b9f9-517d228c399f" containerName="revoke-keys"
Oct 30 03:19:30.754866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.754843    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:30.788156 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.787996    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:19:30.788156 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.788124    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:19:30.790945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.790911    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"]
Oct 30 03:19:30.873211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.873164    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvssw\" (UniqueName: \"kubernetes.io/projected/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-kube-api-access-fvssw\") pod \"maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:30.873448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.873363    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:30.873553 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.873523    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:30.975336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.975288    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvssw\" (UniqueName: \"kubernetes.io/projected/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-kube-api-access-fvssw\") pod \"maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:30.975597 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.975492    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:30.975656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.975606    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:30.976258 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.976230    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:30.978060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.978035    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:30.998309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:30.998264    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvssw\" (UniqueName: \"kubernetes.io/projected/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-kube-api-access-fvssw\") pod \"maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:31.093019 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:31.092696    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:31.553763 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:31.553735    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"]
Oct 30 03:19:31.555514 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:19:31.555472    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5b12ea_7279_4432_88e2_4d6d4e91c7cd.slice/crio-9ef969b520fa0741e31ffbe314e886789922ba4d9362c9704168d6d4b55d1b68 WatchSource:0}: Error finding container 9ef969b520fa0741e31ffbe314e886789922ba4d9362c9704168d6d4b55d1b68: Status 404 returned error can't find the container with id 9ef969b520fa0741e31ffbe314e886789922ba4d9362c9704168d6d4b55d1b68
Oct 30 03:19:32.011201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:32.011163    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n" event={"ID":"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd","Type":"ContainerStarted","Data":"864cb6c8065b638e2aa30e37ff20b7d6f40a96ae0d72132f03bf9613131a71df"}
Oct 30 03:19:32.011665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:32.011223    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n" event={"ID":"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd","Type":"ContainerStarted","Data":"9ef969b520fa0741e31ffbe314e886789922ba4d9362c9704168d6d4b55d1b68"}
Oct 30 03:19:32.054393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:32.054321    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n" podStartSLOduration=2.05430602 podStartE2EDuration="2.05430602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:32.050831248 +0000 UTC m=+2309.975039436" watchObservedRunningTime="2026-10-30 03:19:32.05430602 +0000 UTC m=+2309.978514207"
Oct 30 03:19:33.022755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:33.022701    4278 generic.go:356] "Generic (PLEG): container finished" podID="8c5b12ea-7279-4432-88e2-4d6d4e91c7cd" containerID="864cb6c8065b638e2aa30e37ff20b7d6f40a96ae0d72132f03bf9613131a71df" exitCode=0
Oct 30 03:19:33.023248 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:33.022780    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n" event={"ID":"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd","Type":"ContainerDied","Data":"864cb6c8065b638e2aa30e37ff20b7d6f40a96ae0d72132f03bf9613131a71df"}
Oct 30 03:19:34.432658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.432630    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:34.487666 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.487619    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-xks-ca\") pod \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") "
Oct 30 03:19:34.487859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.487722    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-openshift-ca\") pod \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") "
Oct 30 03:19:34.487962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.487944    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-kube-api-access-fvssw\" (UniqueName: \"kubernetes.io/projected/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-kube-api-access-fvssw\") pod \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\" (UID: \"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd\") "
Oct 30 03:19:34.488193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.488168    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-openshift-ca" pod "8c5b12ea-7279-4432-88e2-4d6d4e91c7cd" (UID: "8c5b12ea-7279-4432-88e2-4d6d4e91c7cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:34.488702 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.488667    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:34.490345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.490315    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-xks-ca" pod "8c5b12ea-7279-4432-88e2-4d6d4e91c7cd" (UID: "8c5b12ea-7279-4432-88e2-4d6d4e91c7cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:34.491481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.491415    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-kube-api-access-fvssw" pod "8c5b12ea-7279-4432-88e2-4d6d4e91c7cd" (UID: "8c5b12ea-7279-4432-88e2-4d6d4e91c7cd"). InnerVolumeSpecName "kube-api-access-fvssw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:34.602371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.599016    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvssw\" (UniqueName: \"kubernetes.io/projected/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-kube-api-access-fvssw\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:34.602371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.599058    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:34.626899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.626864    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"]
Oct 30 03:19:34.633084 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:34.633046    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f21058-9z26j"]
Oct 30 03:19:35.075067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.075038    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"
Oct 30 03:19:35.075288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.075065    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n" event={"ID":"8c5b12ea-7279-4432-88e2-4d6d4e91c7cd","Type":"ContainerDied","Data":"9ef969b520fa0741e31ffbe314e886789922ba4d9362c9704168d6d4b55d1b68"}
Oct 30 03:19:35.075288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.075115    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ef969b520fa0741e31ffbe314e886789922ba4d9362c9704168d6d4b55d1b68"
Oct 30 03:19:35.211556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.211520    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"]
Oct 30 03:19:35.212169 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.212156    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8c5b12ea-7279-4432-88e2-4d6d4e91c7cd" containerName="revoke-keys"
Oct 30 03:19:35.212228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.212171    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c5b12ea-7279-4432-88e2-4d6d4e91c7cd" containerName="revoke-keys"
Oct 30 03:19:35.212319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.212310    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c5b12ea-7279-4432-88e2-4d6d4e91c7cd" containerName="revoke-keys"
Oct 30 03:19:35.216025 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.216003    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.235843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.232293    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:19:35.235843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.232371    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:19:35.236457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.236406    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"]
Oct 30 03:19:35.309996 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.309952    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd0a039-bb5mn\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.310230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.310037    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9jhg\" (UniqueName: \"kubernetes.io/projected/95cae793-895e-4bb0-b884-443c42d4b0f2-kube-api-access-v9jhg\") pod \"maas-api-revoke-sub-dbd0a039-bb5mn\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.310230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.310155    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd0a039-bb5mn\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.411945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.411860    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd0a039-bb5mn\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.411945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.411942    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jhg\" (UniqueName: \"kubernetes.io/projected/95cae793-895e-4bb0-b884-443c42d4b0f2-kube-api-access-v9jhg\") pod \"maas-api-revoke-sub-dbd0a039-bb5mn\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.412288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.411997    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd0a039-bb5mn\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.412624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.412595    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd0a039-bb5mn\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.414349 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.414326    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd0a039-bb5mn\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.432107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.432079    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jhg\" (UniqueName: \"kubernetes.io/projected/95cae793-895e-4bb0-b884-443c42d4b0f2-kube-api-access-v9jhg\") pod \"maas-api-revoke-sub-dbd0a039-bb5mn\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.530754 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.530721    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:35.840034 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:19:35.840003    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cae793_895e_4bb0_b884_443c42d4b0f2.slice/crio-a2bfe43544e71401392f4769484bea2c136833a982928f5b181ab9fc7b68e858 WatchSource:0}: Error finding container a2bfe43544e71401392f4769484bea2c136833a982928f5b181ab9fc7b68e858: Status 404 returned error can't find the container with id a2bfe43544e71401392f4769484bea2c136833a982928f5b181ab9fc7b68e858
Oct 30 03:19:35.840183 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:35.840057    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"]
Oct 30 03:19:36.085347 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:36.085310    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn" event={"ID":"95cae793-895e-4bb0-b884-443c42d4b0f2","Type":"ContainerStarted","Data":"a2bfe43544e71401392f4769484bea2c136833a982928f5b181ab9fc7b68e858"}
Oct 30 03:19:36.201782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:36.201743    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d" path="/var/lib/kubelet/pods/c6d1dd7f-33eb-4cad-ac8d-e82949faeb0d/volumes"
Oct 30 03:19:37.022127 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:19:37.022081    4278 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cae793_895e_4bb0_b884_443c42d4b0f2.slice/crio-conmon-5fa54f090ad0b26906c94863abb2a739e113dd0e74c2a31799dd4280d22f92ee.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cae793_895e_4bb0_b884_443c42d4b0f2.slice/crio-5fa54f090ad0b26906c94863abb2a739e113dd0e74c2a31799dd4280d22f92ee.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:19:37.094988 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:37.094951    4278 generic.go:356] "Generic (PLEG): container finished" podID="95cae793-895e-4bb0-b884-443c42d4b0f2" containerID="5fa54f090ad0b26906c94863abb2a739e113dd0e74c2a31799dd4280d22f92ee" exitCode=0
Oct 30 03:19:37.095206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:37.095039    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn" event={"ID":"95cae793-895e-4bb0-b884-443c42d4b0f2","Type":"ContainerDied","Data":"5fa54f090ad0b26906c94863abb2a739e113dd0e74c2a31799dd4280d22f92ee"}
Oct 30 03:19:38.290809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:38.290783    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:38.361794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:38.361695    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-openshift-ca\") pod \"95cae793-895e-4bb0-b884-443c42d4b0f2\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") "
Oct 30 03:19:38.361987 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:38.361947    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95cae793-895e-4bb0-b884-443c42d4b0f2-kube-api-access-v9jhg\" (UniqueName: \"kubernetes.io/projected/95cae793-895e-4bb0-b884-443c42d4b0f2-kube-api-access-v9jhg\") pod \"95cae793-895e-4bb0-b884-443c42d4b0f2\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") "
Oct 30 03:19:38.362057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:38.362034    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-xks-ca\") pod \"95cae793-895e-4bb0-b884-443c42d4b0f2\" (UID: \"95cae793-895e-4bb0-b884-443c42d4b0f2\") "
Oct 30 03:19:38.362173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:38.362119    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-openshift-ca" pod "95cae793-895e-4bb0-b884-443c42d4b0f2" (UID: "95cae793-895e-4bb0-b884-443c42d4b0f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:38.362778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:38.362737    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:38.364880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:38.364848    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-xks-ca" pod "95cae793-895e-4bb0-b884-443c42d4b0f2" (UID: "95cae793-895e-4bb0-b884-443c42d4b0f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:38.364996 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:38.364960    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95cae793-895e-4bb0-b884-443c42d4b0f2-kube-api-access-v9jhg" pod "95cae793-895e-4bb0-b884-443c42d4b0f2" (UID: "95cae793-895e-4bb0-b884-443c42d4b0f2"). InnerVolumeSpecName "kube-api-access-v9jhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:38.464143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:38.464106    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9jhg\" (UniqueName: \"kubernetes.io/projected/95cae793-895e-4bb0-b884-443c42d4b0f2-kube-api-access-v9jhg\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:38.464143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:38.464140    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95cae793-895e-4bb0-b884-443c42d4b0f2-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:39.116126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:39.116096    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"
Oct 30 03:19:39.116319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:39.116129    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn" event={"ID":"95cae793-895e-4bb0-b884-443c42d4b0f2","Type":"ContainerDied","Data":"a2bfe43544e71401392f4769484bea2c136833a982928f5b181ab9fc7b68e858"}
Oct 30 03:19:39.116319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:39.116189    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2bfe43544e71401392f4769484bea2c136833a982928f5b181ab9fc7b68e858"
Oct 30 03:19:41.312623 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.312583    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"]
Oct 30 03:19:41.313137 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.312843    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerName="istio-proxy" containerID="cri-o://3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34" gracePeriod=30
Oct 30 03:19:41.819059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.819023    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"]
Oct 30 03:19:41.819787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.819767    4278 cpu_manager.go:414] "containerMap: removing container" podUID="95cae793-895e-4bb0-b884-443c42d4b0f2" containerName="revoke-keys"
Oct 30 03:19:41.819787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.819788    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cae793-895e-4bb0-b884-443c42d4b0f2" containerName="revoke-keys"
Oct 30 03:19:41.819936 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.819927    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cae793-895e-4bb0-b884-443c42d4b0f2" containerName="revoke-keys"
Oct 30 03:19:41.823975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.823955    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:41.843301 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.843269    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"]
Oct 30 03:19:41.884503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.873830    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:19:41.884503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.873984    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:19:41.941489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.937270    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l76bl\" (UniqueName: \"kubernetes.io/projected/8bbc97af-8e18-43a4-8237-6f1fb339306c-kube-api-access-l76bl\") pod \"maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:41.941489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.937383    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:41.941489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:41.937489    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:42.039588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:42.039554    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l76bl\" (UniqueName: \"kubernetes.io/projected/8bbc97af-8e18-43a4-8237-6f1fb339306c-kube-api-access-l76bl\") pod \"maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:42.039849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:42.039649    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:42.039849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:42.039739    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:42.040499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:42.040476    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:42.043099 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:42.043072    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:42.060737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:42.060700    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l76bl\" (UniqueName: \"kubernetes.io/projected/8bbc97af-8e18-43a4-8237-6f1fb339306c-kube-api-access-l76bl\") pod \"maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:42.141537 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:42.140910    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:42.468662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:42.468626    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"]
Oct 30 03:19:42.469176 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:19:42.469151    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbc97af_8e18_43a4_8237_6f1fb339306c.slice/crio-663b0d61ca255d1d8781b370f7731016408a4c0e4bad6d3a084e1eb4243d7ce2 WatchSource:0}: Error finding container 663b0d61ca255d1d8781b370f7731016408a4c0e4bad6d3a084e1eb4243d7ce2: Status 404 returned error can't find the container with id 663b0d61ca255d1d8781b370f7731016408a4c0e4bad6d3a084e1eb4243d7ce2
Oct 30 03:19:43.154065 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:43.153964    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz" event={"ID":"8bbc97af-8e18-43a4-8237-6f1fb339306c","Type":"ContainerStarted","Data":"ac4fe84b0c99a6e2e39e40cf48671dd0c0c1812919b690b4bcfee7b4004af350"}
Oct 30 03:19:43.154065 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:43.154053    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz" event={"ID":"8bbc97af-8e18-43a4-8237-6f1fb339306c","Type":"ContainerStarted","Data":"663b0d61ca255d1d8781b370f7731016408a4c0e4bad6d3a084e1eb4243d7ce2"}
Oct 30 03:19:43.197036 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:43.196977    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz" podStartSLOduration=2.19695828 podStartE2EDuration="2.19695828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:43.192249993 +0000 UTC m=+2321.116458184" watchObservedRunningTime="2026-10-30 03:19:43.19695828 +0000 UTC m=+2321.121166471"
Oct 30 03:19:44.164271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:44.164236    4278 generic.go:356] "Generic (PLEG): container finished" podID="8bbc97af-8e18-43a4-8237-6f1fb339306c" containerID="ac4fe84b0c99a6e2e39e40cf48671dd0c0c1812919b690b4bcfee7b4004af350" exitCode=0
Oct 30 03:19:44.164715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:44.164326    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz" event={"ID":"8bbc97af-8e18-43a4-8237-6f1fb339306c","Type":"ContainerDied","Data":"ac4fe84b0c99a6e2e39e40cf48671dd0c0c1812919b690b4bcfee7b4004af350"}
Oct 30 03:19:45.344769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:45.344744    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:45.390259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:45.390218    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-openshift-ca\") pod \"8bbc97af-8e18-43a4-8237-6f1fb339306c\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") "
Oct 30 03:19:45.390574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:45.390551    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bbc97af-8e18-43a4-8237-6f1fb339306c-kube-api-access-l76bl\" (UniqueName: \"kubernetes.io/projected/8bbc97af-8e18-43a4-8237-6f1fb339306c-kube-api-access-l76bl\") pod \"8bbc97af-8e18-43a4-8237-6f1fb339306c\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") "
Oct 30 03:19:45.390677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:45.390632    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-xks-ca\") pod \"8bbc97af-8e18-43a4-8237-6f1fb339306c\" (UID: \"8bbc97af-8e18-43a4-8237-6f1fb339306c\") "
Oct 30 03:19:45.390740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:45.390679    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-openshift-ca" pod "8bbc97af-8e18-43a4-8237-6f1fb339306c" (UID: "8bbc97af-8e18-43a4-8237-6f1fb339306c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:45.391237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:45.391216    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:45.392785 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:45.392762    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-xks-ca" pod "8bbc97af-8e18-43a4-8237-6f1fb339306c" (UID: "8bbc97af-8e18-43a4-8237-6f1fb339306c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:45.393102 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:45.393084    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbc97af-8e18-43a4-8237-6f1fb339306c-kube-api-access-l76bl" pod "8bbc97af-8e18-43a4-8237-6f1fb339306c" (UID: "8bbc97af-8e18-43a4-8237-6f1fb339306c"). InnerVolumeSpecName "kube-api-access-l76bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:45.493500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:45.493389    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l76bl\" (UniqueName: \"kubernetes.io/projected/8bbc97af-8e18-43a4-8237-6f1fb339306c-kube-api-access-l76bl\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:45.493500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:45.493450    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbc97af-8e18-43a4-8237-6f1fb339306c-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.014484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.012120    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:19:46.180758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.180720    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz" event={"ID":"8bbc97af-8e18-43a4-8237-6f1fb339306c","Type":"ContainerDied","Data":"663b0d61ca255d1d8781b370f7731016408a4c0e4bad6d3a084e1eb4243d7ce2"}
Oct 30 03:19:46.180758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.180753    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"
Oct 30 03:19:46.180974 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.180772    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="663b0d61ca255d1d8781b370f7731016408a4c0e4bad6d3a084e1eb4243d7ce2"
Oct 30 03:19:46.297857 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.297769    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"]
Oct 30 03:19:46.298560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.298538    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8bbc97af-8e18-43a4-8237-6f1fb339306c" containerName="revoke-keys"
Oct 30 03:19:46.298560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.298560    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bbc97af-8e18-43a4-8237-6f1fb339306c" containerName="revoke-keys"
Oct 30 03:19:46.298824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.298730    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bbc97af-8e18-43a4-8237-6f1fb339306c" containerName="revoke-keys"
Oct 30 03:19:46.302828 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.302803    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.335169 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.335140    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:19:46.337101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.337074    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:19:46.342293 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.342265    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"]
Oct 30 03:19:46.412439 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.412394    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268cba7c-xd94d\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.412829 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.412691    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268cba7c-xd94d\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.412872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.412817    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbvv\" (UniqueName: \"kubernetes.io/projected/518cf17e-ff4c-40c8-b88f-c98413289657-kube-api-access-nzbvv\") pod \"maas-api-revoke-sub-268cba7c-xd94d\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.514232 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.514094    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268cba7c-xd94d\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.514232 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.514203    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbvv\" (UniqueName: \"kubernetes.io/projected/518cf17e-ff4c-40c8-b88f-c98413289657-kube-api-access-nzbvv\") pod \"maas-api-revoke-sub-268cba7c-xd94d\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.514521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.514258    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268cba7c-xd94d\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.515195 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.515163    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268cba7c-xd94d\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.516897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.516865    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268cba7c-xd94d\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.535984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.535954    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbvv\" (UniqueName: \"kubernetes.io/projected/518cf17e-ff4c-40c8-b88f-c98413289657-kube-api-access-nzbvv\") pod \"maas-api-revoke-sub-268cba7c-xd94d\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.620962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.620935    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:46.635461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.635418    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:19:46.716377 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.716337    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-kube-api-access-98b7q\" (UniqueName: \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-kube-api-access-98b7q\") pod \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") "
Oct 30 03:19:46.716581 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.716442    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istiod-ca-cert\") pod \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") "
Oct 30 03:19:46.716581 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.716540    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-socket\") pod \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") "
Oct 30 03:19:46.716668 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.716623    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-podinfo\") pod \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") "
Oct 30 03:19:46.716713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.716663    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-envoy\") pod \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") "
Oct 30 03:19:46.716713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.716704    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-token\" (UniqueName: \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-token\") pod \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") "
Oct 30 03:19:46.716791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.716752    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-certs\") pod \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") "
Oct 30 03:19:46.716825 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.716807    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-credential-socket\") pod \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") "
Oct 30 03:19:46.716866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.716856    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-data\") pod \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\" (UID: \"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b\") "
Oct 30 03:19:46.718068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.718040    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-data" pod "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" (UID: "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:46.718720 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.718691    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-socket" pod "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" (UID: "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:46.719041 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.719017    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istiod-ca-cert" pod "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" (UID: "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:46.721383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.721354    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-podinfo" pod "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" (UID: "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:19:46.721527 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.721395    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-certs" pod "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" (UID: "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:46.721666 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.721647    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-credential-socket" pod "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" (UID: "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:46.721774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.721759    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-kube-api-access-98b7q" pod "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" (UID: "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b"). InnerVolumeSpecName "kube-api-access-98b7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:46.726644 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.726618    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-envoy" pod "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" (UID: "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:46.726744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.726646    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-token" pod "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" (UID: "090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:46.819060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.819019    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98b7q\" (UniqueName: \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-kube-api-access-98b7q\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.819060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.819055    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.819060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.819066    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.819305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.819075    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.819305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.819086    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.819305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.819094    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.819305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.819101    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.819305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.819109    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.819305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.819118    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.931592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:46.931564    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"]
Oct 30 03:19:46.931737 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:19:46.931712    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518cf17e_ff4c_40c8_b88f_c98413289657.slice/crio-f0ff707525a4fbc6cf443e224b88bb216328e565fc74a4b48b054de2239f2568 WatchSource:0}: Error finding container f0ff707525a4fbc6cf443e224b88bb216328e565fc74a4b48b054de2239f2568: Status 404 returned error can't find the container with id f0ff707525a4fbc6cf443e224b88bb216328e565fc74a4b48b054de2239f2568
Oct 30 03:19:47.200508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:47.200212    4278 generic.go:356] "Generic (PLEG): container finished" podID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerID="3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34" exitCode=0
Oct 30 03:19:47.200697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:47.200508    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"
Oct 30 03:19:47.214471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:47.203618    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" event={"ID":"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b","Type":"ContainerDied","Data":"3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34"}
Oct 30 03:19:47.214471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:47.203679    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg" event={"ID":"090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b","Type":"ContainerDied","Data":"c0fa19bcbb3e4d864156cd41b4ea849b22236e73b9c079ab266ccd9ae7af9395"}
Oct 30 03:19:47.214471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:47.203708    4278 scope.go:122] "RemoveContainer" containerID="3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34"
Oct 30 03:19:47.214471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:47.208300    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d" event={"ID":"518cf17e-ff4c-40c8-b88f-c98413289657","Type":"ContainerStarted","Data":"f0ff707525a4fbc6cf443e224b88bb216328e565fc74a4b48b054de2239f2568"}
Oct 30 03:19:47.229384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:47.226953    4278 scope.go:122] "RemoveContainer" containerID="3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34"
Oct 30 03:19:47.229384 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:19:47.227349    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34\": container with ID starting with 3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34 not found: ID does not exist" containerID="3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34"
Oct 30 03:19:47.229384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:47.227391    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34"} err="failed to get container status \"3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34\": rpc error: code = NotFound desc = could not find container \"3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34\": container with ID starting with 3c2b2485deafa18dba45f8fe7d7b7d7284608fee0c1d251c0ec98113f9fb4d34 not found: ID does not exist"
Oct 30 03:19:47.277569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:47.277532    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"]
Oct 30 03:19:47.294062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:47.294012    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-rlbcg"]
Oct 30 03:19:48.248349 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:48.248310    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" path="/var/lib/kubelet/pods/090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b/volumes"
Oct 30 03:19:48.255838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:48.255810    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d" event={"ID":"518cf17e-ff4c-40c8-b88f-c98413289657","Type":"ContainerStarted","Data":"8e96d57b2cf66a1cca0498df9f516fc317b274ad0516e1b86407ec4cd0f04d93"}
Oct 30 03:19:48.309447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:48.299483    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d" podStartSLOduration=2.299466707 podStartE2EDuration="2.299466707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:48.295864653 +0000 UTC m=+2326.220072845" watchObservedRunningTime="2026-10-30 03:19:48.299466707 +0000 UTC m=+2326.223674938"
Oct 30 03:19:49.265677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:49.265638    4278 generic.go:356] "Generic (PLEG): container finished" podID="518cf17e-ff4c-40c8-b88f-c98413289657" containerID="8e96d57b2cf66a1cca0498df9f516fc317b274ad0516e1b86407ec4cd0f04d93" exitCode=0
Oct 30 03:19:49.266140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:49.265725    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d" event={"ID":"518cf17e-ff4c-40c8-b88f-c98413289657","Type":"ContainerDied","Data":"8e96d57b2cf66a1cca0498df9f516fc317b274ad0516e1b86407ec4cd0f04d93"}
Oct 30 03:19:50.485015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:50.484978    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:50.517338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:50.517292    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518cf17e-ff4c-40c8-b88f-c98413289657-kube-api-access-nzbvv\" (UniqueName: \"kubernetes.io/projected/518cf17e-ff4c-40c8-b88f-c98413289657-kube-api-access-nzbvv\") pod \"518cf17e-ff4c-40c8-b88f-c98413289657\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") "
Oct 30 03:19:50.517561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:50.517509    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-xks-ca\") pod \"518cf17e-ff4c-40c8-b88f-c98413289657\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") "
Oct 30 03:19:50.517825 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:50.517648    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-openshift-ca\") pod \"518cf17e-ff4c-40c8-b88f-c98413289657\" (UID: \"518cf17e-ff4c-40c8-b88f-c98413289657\") "
Oct 30 03:19:50.518089 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:50.518059    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-openshift-ca" pod "518cf17e-ff4c-40c8-b88f-c98413289657" (UID: "518cf17e-ff4c-40c8-b88f-c98413289657"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:50.518506 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:50.518483    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:50.520006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:50.519979    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518cf17e-ff4c-40c8-b88f-c98413289657-kube-api-access-nzbvv" pod "518cf17e-ff4c-40c8-b88f-c98413289657" (UID: "518cf17e-ff4c-40c8-b88f-c98413289657"). InnerVolumeSpecName "kube-api-access-nzbvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:50.520146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:50.520120    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-xks-ca" pod "518cf17e-ff4c-40c8-b88f-c98413289657" (UID: "518cf17e-ff4c-40c8-b88f-c98413289657"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:50.620387 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:50.620350    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518cf17e-ff4c-40c8-b88f-c98413289657-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:50.620387 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:50.620391    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbvv\" (UniqueName: \"kubernetes.io/projected/518cf17e-ff4c-40c8-b88f-c98413289657-kube-api-access-nzbvv\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:19:51.321020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:51.320977    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d" event={"ID":"518cf17e-ff4c-40c8-b88f-c98413289657","Type":"ContainerDied","Data":"f0ff707525a4fbc6cf443e224b88bb216328e565fc74a4b48b054de2239f2568"}
Oct 30 03:19:51.321230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:51.321043    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"
Oct 30 03:19:51.321230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:51.321068    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0ff707525a4fbc6cf443e224b88bb216328e565fc74a4b48b054de2239f2568"
Oct 30 03:19:54.192577 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:19:54.192541    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:19:56.880697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.880661    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"]
Oct 30 03:19:56.881531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.881504    4278 cpu_manager.go:414] "containerMap: removing container" podUID="518cf17e-ff4c-40c8-b88f-c98413289657" containerName="revoke-keys"
Oct 30 03:19:56.881531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.881526    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518cf17e-ff4c-40c8-b88f-c98413289657" containerName="revoke-keys"
Oct 30 03:19:56.881821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.881575    4278 cpu_manager.go:414] "containerMap: removing container" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerName="istio-proxy"
Oct 30 03:19:56.881821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.881581    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerName="istio-proxy"
Oct 30 03:19:56.881821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.881750    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="090d3f9c-73ac-4bf1-90e2-e0fc9e9db43b" containerName="istio-proxy"
Oct 30 03:19:56.881821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.881772    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="518cf17e-ff4c-40c8-b88f-c98413289657" containerName="revoke-keys"
Oct 30 03:19:56.887242 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.887215    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:56.914512 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.907198    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"]
Oct 30 03:19:56.916783 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.916747    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:19:56.916938 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.916909    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:19:56.932944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.932895    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdac2ce7-r6kgt\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:56.933151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.933128    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdac2ce7-r6kgt\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:56.933216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:56.933190    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvlbv\" (UniqueName: \"kubernetes.io/projected/da6143d2-4417-4f00-97b0-cd796d221616-kube-api-access-kvlbv\") pod \"maas-api-revoke-sub-bdac2ce7-r6kgt\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:57.034421 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:57.034383    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdac2ce7-r6kgt\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:57.034613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:57.034519    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvlbv\" (UniqueName: \"kubernetes.io/projected/da6143d2-4417-4f00-97b0-cd796d221616-kube-api-access-kvlbv\") pod \"maas-api-revoke-sub-bdac2ce7-r6kgt\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:57.034773 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:57.034757    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdac2ce7-r6kgt\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:57.035539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:57.035508    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdac2ce7-r6kgt\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:57.037093 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:57.037071    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdac2ce7-r6kgt\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:57.054711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:57.054682    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvlbv\" (UniqueName: \"kubernetes.io/projected/da6143d2-4417-4f00-97b0-cd796d221616-kube-api-access-kvlbv\") pod \"maas-api-revoke-sub-bdac2ce7-r6kgt\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:57.211133 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:57.211047    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:19:57.676498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:57.676419    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"]
Oct 30 03:19:57.677846 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:19:57.677816    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6143d2_4417_4f00_97b0_cd796d221616.slice/crio-3a2d5c8c1999b25a48c789238f4f043af93edd96b626640d7d0c04a8a8dba5f6 WatchSource:0}: Error finding container 3a2d5c8c1999b25a48c789238f4f043af93edd96b626640d7d0c04a8a8dba5f6: Status 404 returned error can't find the container with id 3a2d5c8c1999b25a48c789238f4f043af93edd96b626640d7d0c04a8a8dba5f6
Oct 30 03:19:58.417538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:58.417503    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt" event={"ID":"da6143d2-4417-4f00-97b0-cd796d221616","Type":"ContainerStarted","Data":"baec05321b320b792329f2035f4009274cc8370fed381afc8b86104f17cbba78"}
Oct 30 03:19:58.417980 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:58.417562    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt" event={"ID":"da6143d2-4417-4f00-97b0-cd796d221616","Type":"ContainerStarted","Data":"3a2d5c8c1999b25a48c789238f4f043af93edd96b626640d7d0c04a8a8dba5f6"}
Oct 30 03:19:58.459573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:58.459515    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt" podStartSLOduration=2.459492256 podStartE2EDuration="2.459492256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:58.456251408 +0000 UTC m=+2336.380459595" watchObservedRunningTime="2026-10-30 03:19:58.459492256 +0000 UTC m=+2336.383700505"
Oct 30 03:19:59.426905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:59.426866    4278 generic.go:356] "Generic (PLEG): container finished" podID="da6143d2-4417-4f00-97b0-cd796d221616" containerID="baec05321b320b792329f2035f4009274cc8370fed381afc8b86104f17cbba78" exitCode=0
Oct 30 03:19:59.427358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:19:59.426957    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt" event={"ID":"da6143d2-4417-4f00-97b0-cd796d221616","Type":"ContainerDied","Data":"baec05321b320b792329f2035f4009274cc8370fed381afc8b86104f17cbba78"}
Oct 30 03:20:00.637995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:00.637966    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:20:00.725966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:00.725867    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-openshift-ca\") pod \"da6143d2-4417-4f00-97b0-cd796d221616\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") "
Oct 30 03:20:00.726143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:00.726025    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-xks-ca\") pod \"da6143d2-4417-4f00-97b0-cd796d221616\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") "
Oct 30 03:20:00.726143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:00.726096    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6143d2-4417-4f00-97b0-cd796d221616-kube-api-access-kvlbv\" (UniqueName: \"kubernetes.io/projected/da6143d2-4417-4f00-97b0-cd796d221616-kube-api-access-kvlbv\") pod \"da6143d2-4417-4f00-97b0-cd796d221616\" (UID: \"da6143d2-4417-4f00-97b0-cd796d221616\") "
Oct 30 03:20:00.726388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:00.726355    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-openshift-ca" pod "da6143d2-4417-4f00-97b0-cd796d221616" (UID: "da6143d2-4417-4f00-97b0-cd796d221616"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:00.726838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:00.726807    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:00.728438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:00.728396    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-xks-ca" pod "da6143d2-4417-4f00-97b0-cd796d221616" (UID: "da6143d2-4417-4f00-97b0-cd796d221616"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:00.728582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:00.728504    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6143d2-4417-4f00-97b0-cd796d221616-kube-api-access-kvlbv" pod "da6143d2-4417-4f00-97b0-cd796d221616" (UID: "da6143d2-4417-4f00-97b0-cd796d221616"). InnerVolumeSpecName "kube-api-access-kvlbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:00.828725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:00.828693    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6143d2-4417-4f00-97b0-cd796d221616-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:00.828725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:00.828722    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvlbv\" (UniqueName: \"kubernetes.io/projected/da6143d2-4417-4f00-97b0-cd796d221616-kube-api-access-kvlbv\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:01.453400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:01.453362    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"
Oct 30 03:20:01.453400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:01.453392    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt" event={"ID":"da6143d2-4417-4f00-97b0-cd796d221616","Type":"ContainerDied","Data":"3a2d5c8c1999b25a48c789238f4f043af93edd96b626640d7d0c04a8a8dba5f6"}
Oct 30 03:20:01.453672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:01.453485    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a2d5c8c1999b25a48c789238f4f043af93edd96b626640d7d0c04a8a8dba5f6"
Oct 30 03:20:04.194397 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:04.194359    4278 scope.go:122] "RemoveContainer" containerID="2adc41119f83177f95eaed385cbbc44a7551bab729dd68d7231c57e7c40bf78f"
Oct 30 03:20:04.206879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:04.206855    4278 scope.go:122] "RemoveContainer" containerID="f40f720dc848be0abf5e70cca496bfc914962f20fb379a839b7a18ad96b6c7fa"
Oct 30 03:20:09.688932 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:09.688891    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"]
Oct 30 03:20:09.701661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:09.701626    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-100ffd-f9n4n"]
Oct 30 03:20:10.202440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:10.202393    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c5b12ea-7279-4432-88e2-4d6d4e91c7cd" path="/var/lib/kubelet/pods/8c5b12ea-7279-4432-88e2-4d6d4e91c7cd/volumes"
Oct 30 03:20:11.370912 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:11.370873    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"]
Oct 30 03:20:12.789730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.789690    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"]
Oct 30 03:20:12.790395 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.790377    4278 cpu_manager.go:414] "containerMap: removing container" podUID="da6143d2-4417-4f00-97b0-cd796d221616" containerName="revoke-keys"
Oct 30 03:20:12.790456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.790396    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6143d2-4417-4f00-97b0-cd796d221616" containerName="revoke-keys"
Oct 30 03:20:12.790547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.790536    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6143d2-4417-4f00-97b0-cd796d221616" containerName="revoke-keys"
Oct 30 03:20:12.796052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.796029    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:12.817354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.817323    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"]
Oct 30 03:20:12.824638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.824608    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:20:12.825164 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.825125    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:20:12.835243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.835203    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r54w7\" (UniqueName: \"kubernetes.io/projected/0905bcd3-d19d-45e2-99a3-ca80d434452a-kube-api-access-r54w7\") pod \"maas-api-revoke-sub-35143558-c4x45\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:12.835421 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.835403    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35143558-c4x45\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:12.835554 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.835536    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35143558-c4x45\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:12.937731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.937695    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r54w7\" (UniqueName: \"kubernetes.io/projected/0905bcd3-d19d-45e2-99a3-ca80d434452a-kube-api-access-r54w7\") pod \"maas-api-revoke-sub-35143558-c4x45\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:12.937937 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.937799    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35143558-c4x45\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:12.937937 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.937854    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35143558-c4x45\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:12.938579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.938554    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35143558-c4x45\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:12.940534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.940512    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35143558-c4x45\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:12.960685 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:12.960653    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r54w7\" (UniqueName: \"kubernetes.io/projected/0905bcd3-d19d-45e2-99a3-ca80d434452a-kube-api-access-r54w7\") pod \"maas-api-revoke-sub-35143558-c4x45\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:13.112576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:13.112481    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:13.422392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:13.422358    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"]
Oct 30 03:20:13.422901 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:20:13.422878    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0905bcd3_d19d_45e2_99a3_ca80d434452a.slice/crio-1883f5cf4888827df576f4697ef331f2bee236413fbaaf08639a68178b4f71c4 WatchSource:0}: Error finding container 1883f5cf4888827df576f4697ef331f2bee236413fbaaf08639a68178b4f71c4: Status 404 returned error can't find the container with id 1883f5cf4888827df576f4697ef331f2bee236413fbaaf08639a68178b4f71c4
Oct 30 03:20:13.636678 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:13.636632    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45" event={"ID":"0905bcd3-d19d-45e2-99a3-ca80d434452a","Type":"ContainerStarted","Data":"1883f5cf4888827df576f4697ef331f2bee236413fbaaf08639a68178b4f71c4"}
Oct 30 03:20:14.647029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:14.646931    4278 generic.go:356] "Generic (PLEG): container finished" podID="0905bcd3-d19d-45e2-99a3-ca80d434452a" containerID="61c8e493e1565ef457d83ac798949783d96d55ae1b824188c3b9c7d856b4cd0a" exitCode=0
Oct 30 03:20:14.647500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:14.647012    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45" event={"ID":"0905bcd3-d19d-45e2-99a3-ca80d434452a","Type":"ContainerDied","Data":"61c8e493e1565ef457d83ac798949783d96d55ae1b824188c3b9c7d856b4cd0a"}
Oct 30 03:20:15.822663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:15.822624    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:15.883647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:15.883614    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-openshift-ca\") pod \"0905bcd3-d19d-45e2-99a3-ca80d434452a\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") "
Oct 30 03:20:15.883864 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:15.883673    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0905bcd3-d19d-45e2-99a3-ca80d434452a-kube-api-access-r54w7\" (UniqueName: \"kubernetes.io/projected/0905bcd3-d19d-45e2-99a3-ca80d434452a-kube-api-access-r54w7\") pod \"0905bcd3-d19d-45e2-99a3-ca80d434452a\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") "
Oct 30 03:20:15.883951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:15.883923    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-xks-ca\") pod \"0905bcd3-d19d-45e2-99a3-ca80d434452a\" (UID: \"0905bcd3-d19d-45e2-99a3-ca80d434452a\") "
Oct 30 03:20:15.884088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:15.884058    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-openshift-ca" pod "0905bcd3-d19d-45e2-99a3-ca80d434452a" (UID: "0905bcd3-d19d-45e2-99a3-ca80d434452a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:15.884626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:15.884601    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:15.886674 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:15.886644    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0905bcd3-d19d-45e2-99a3-ca80d434452a-kube-api-access-r54w7" pod "0905bcd3-d19d-45e2-99a3-ca80d434452a" (UID: "0905bcd3-d19d-45e2-99a3-ca80d434452a"). InnerVolumeSpecName "kube-api-access-r54w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:15.886674 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:15.886663    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-xks-ca" pod "0905bcd3-d19d-45e2-99a3-ca80d434452a" (UID: "0905bcd3-d19d-45e2-99a3-ca80d434452a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:15.986093 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:15.986001    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0905bcd3-d19d-45e2-99a3-ca80d434452a-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:15.986093 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:15.986037    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r54w7\" (UniqueName: \"kubernetes.io/projected/0905bcd3-d19d-45e2-99a3-ca80d434452a-kube-api-access-r54w7\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:16.664288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:16.664255    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"
Oct 30 03:20:16.664551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:16.664289    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45" event={"ID":"0905bcd3-d19d-45e2-99a3-ca80d434452a","Type":"ContainerDied","Data":"1883f5cf4888827df576f4697ef331f2bee236413fbaaf08639a68178b4f71c4"}
Oct 30 03:20:16.664551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:16.664343    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1883f5cf4888827df576f4697ef331f2bee236413fbaaf08639a68178b4f71c4"
Oct 30 03:20:16.724177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:16.724137    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"]
Oct 30 03:20:16.724715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:16.724673    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy" containerID="cri-o://65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704" gracePeriod=30
Oct 30 03:20:19.316929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:19.316891    4278 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:20:19.317326 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:19.316968    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:20:22.248192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.248160    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:20:22.405867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.405786    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-data\") pod \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") "
Oct 30 03:20:22.405867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.405826    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-socket\") pod \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") "
Oct 30 03:20:22.406107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.405929    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-podinfo\") pod \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") "
Oct 30 03:20:22.406107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406001    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-kube-api-access-qfl5t\" (UniqueName: \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-kube-api-access-qfl5t\") pod \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") "
Oct 30 03:20:22.406107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406042    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-credential-socket\") pod \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") "
Oct 30 03:20:22.406107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406054    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-socket" pod "5da52699-c3f3-420e-9db9-b52af3d7cc6f" (UID: "5da52699-c3f3-420e-9db9-b52af3d7cc6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.406355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406096    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-data" pod "5da52699-c3f3-420e-9db9-b52af3d7cc6f" (UID: "5da52699-c3f3-420e-9db9-b52af3d7cc6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.406355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406084    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-token\" (UniqueName: \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-token\") pod \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") "
Oct 30 03:20:22.406355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406317    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-credential-socket" pod "5da52699-c3f3-420e-9db9-b52af3d7cc6f" (UID: "5da52699-c3f3-420e-9db9-b52af3d7cc6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.406355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406324    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-certs\") pod \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") "
Oct 30 03:20:22.406664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406371    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istiod-ca-cert\") pod \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") "
Oct 30 03:20:22.406664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406567    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-envoy\") pod \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\" (UID: \"5da52699-c3f3-420e-9db9-b52af3d7cc6f\") "
Oct 30 03:20:22.406664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406606    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-certs" pod "5da52699-c3f3-420e-9db9-b52af3d7cc6f" (UID: "5da52699-c3f3-420e-9db9-b52af3d7cc6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.406834 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.406712    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istiod-ca-cert" pod "5da52699-c3f3-420e-9db9-b52af3d7cc6f" (UID: "5da52699-c3f3-420e-9db9-b52af3d7cc6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:22.407353 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.407325    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.407501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.407359    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.407501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.407375    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.407501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.407390    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.407501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.407405    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.408558 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.408530    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-token" pod "5da52699-c3f3-420e-9db9-b52af3d7cc6f" (UID: "5da52699-c3f3-420e-9db9-b52af3d7cc6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:22.408676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.408566    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-kube-api-access-qfl5t" pod "5da52699-c3f3-420e-9db9-b52af3d7cc6f" (UID: "5da52699-c3f3-420e-9db9-b52af3d7cc6f"). InnerVolumeSpecName "kube-api-access-qfl5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:22.408676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.408563    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-podinfo" pod "5da52699-c3f3-420e-9db9-b52af3d7cc6f" (UID: "5da52699-c3f3-420e-9db9-b52af3d7cc6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:20:22.408777 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.408681    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-envoy" pod "5da52699-c3f3-420e-9db9-b52af3d7cc6f" (UID: "5da52699-c3f3-420e-9db9-b52af3d7cc6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.509245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.509183    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.509245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.509227    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.509245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.509242    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da52699-c3f3-420e-9db9-b52af3d7cc6f-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.509664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.509257    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfl5t\" (UniqueName: \"kubernetes.io/projected/5da52699-c3f3-420e-9db9-b52af3d7cc6f-kube-api-access-qfl5t\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.511819 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.511787    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"]
Oct 30 03:20:22.535382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.535346    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"]
Oct 30 03:20:22.541588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.541554    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2305f4-zkjmz"]
Oct 30 03:20:22.771884 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.771848    4278 generic.go:356] "Generic (PLEG): container finished" podID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerID="65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704" exitCode=0
Oct 30 03:20:22.772183 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.772157    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"
Oct 30 03:20:22.772501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.772149    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" event={"ID":"5da52699-c3f3-420e-9db9-b52af3d7cc6f","Type":"ContainerDied","Data":"65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704"}
Oct 30 03:20:22.773259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.773230    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84" event={"ID":"5da52699-c3f3-420e-9db9-b52af3d7cc6f","Type":"ContainerDied","Data":"42d6aa4899893b2a124144ac36edd38a683b33223ba2d93a07927fb4112b99a2"}
Oct 30 03:20:22.773453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.773395    4278 scope.go:122] "RemoveContainer" containerID="65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704"
Oct 30 03:20:22.805503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.805472    4278 scope.go:122] "RemoveContainer" containerID="65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704"
Oct 30 03:20:22.806025 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:20:22.805949    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704\": container with ID starting with 65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704 not found: ID does not exist" containerID="65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704"
Oct 30 03:20:22.806176 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.806037    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704"} err="failed to get container status \"65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704\": rpc error: code = NotFound desc = could not find container \"65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704\": container with ID starting with 65ce89a9bd9f696dc3a75e03aca8ca930a82e993645afac3cd18830ebf7c7704 not found: ID does not exist"
Oct 30 03:20:22.838138 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.838102    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"]
Oct 30 03:20:22.842444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:22.842397    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-tdz84"]
Oct 30 03:20:24.196810 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:20:24.196770    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:20:24.203580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:24.203544    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" path="/var/lib/kubelet/pods/5da52699-c3f3-420e-9db9-b52af3d7cc6f/volumes"
Oct 30 03:20:24.204468 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:24.204400    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bbc97af-8e18-43a4-8237-6f1fb339306c" path="/var/lib/kubelet/pods/8bbc97af-8e18-43a4-8237-6f1fb339306c/volumes"
Oct 30 03:20:25.746806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:25.746763    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"]
Oct 30 03:20:26.335066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.335021    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"]
Oct 30 03:20:26.336524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.336495    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy"
Oct 30 03:20:26.336524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.336523    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy"
Oct 30 03:20:26.336772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.336597    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0905bcd3-d19d-45e2-99a3-ca80d434452a" containerName="revoke-keys"
Oct 30 03:20:26.336772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.336605    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0905bcd3-d19d-45e2-99a3-ca80d434452a" containerName="revoke-keys"
Oct 30 03:20:26.337014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.336987    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da52699-c3f3-420e-9db9-b52af3d7cc6f" containerName="istio-proxy"
Oct 30 03:20:26.337152 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.337026    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0905bcd3-d19d-45e2-99a3-ca80d434452a" containerName="revoke-keys"
Oct 30 03:20:26.350636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.350603    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.360489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.360457    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"]
Oct 30 03:20:26.365881 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.365844    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:20:26.366051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.365976    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:20:26.397472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.393469    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm" podUID="b4f9f398-a946-403d-89ff-82a803d6f639" containerName="main" containerID="cri-o://2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3" gracePeriod=45
Oct 30 03:20:26.463684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.463633    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3406363b-dwhh7\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.463895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.463808    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wkk\" (UniqueName: \"kubernetes.io/projected/0bafe140-0707-40d7-ac96-302a8d01be97-kube-api-access-v7wkk\") pod \"maas-api-revoke-sub-3406363b-dwhh7\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.464008 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.463980    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3406363b-dwhh7\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.565694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.565647    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3406363b-dwhh7\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.565882 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.565779    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wkk\" (UniqueName: \"kubernetes.io/projected/0bafe140-0707-40d7-ac96-302a8d01be97-kube-api-access-v7wkk\") pod \"maas-api-revoke-sub-3406363b-dwhh7\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.565941 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.565904    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3406363b-dwhh7\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.566615 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.566584    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3406363b-dwhh7\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.568809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.568779    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3406363b-dwhh7\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.587301 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.587221    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wkk\" (UniqueName: \"kubernetes.io/projected/0bafe140-0707-40d7-ac96-302a8d01be97-kube-api-access-v7wkk\") pod \"maas-api-revoke-sub-3406363b-dwhh7\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.673823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.673788    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:26.766381 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.766349    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:20:26.872500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.871818    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-model-cache\") pod \"b4f9f398-a946-403d-89ff-82a803d6f639\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") "
Oct 30 03:20:26.872500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.871893    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-home\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-home\") pod \"b4f9f398-a946-403d-89ff-82a803d6f639\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") "
Oct 30 03:20:26.872500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.871989    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f9f398-a946-403d-89ff-82a803d6f639-kube-api-access-84vj9\" (UniqueName: \"kubernetes.io/projected/b4f9f398-a946-403d-89ff-82a803d6f639-kube-api-access-84vj9\") pod \"b4f9f398-a946-403d-89ff-82a803d6f639\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") "
Oct 30 03:20:26.872500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.872197    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-tmp-dir\") pod \"b4f9f398-a946-403d-89ff-82a803d6f639\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") "
Oct 30 03:20:26.872500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.872220    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-model-cache" pod "b4f9f398-a946-403d-89ff-82a803d6f639" (UID: "b4f9f398-a946-403d-89ff-82a803d6f639"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:26.872500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.872366    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-dshm\") pod \"b4f9f398-a946-403d-89ff-82a803d6f639\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") "
Oct 30 03:20:26.872500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.872400    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f9f398-a946-403d-89ff-82a803d6f639-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f9f398-a946-403d-89ff-82a803d6f639-tls-certs\") pod \"b4f9f398-a946-403d-89ff-82a803d6f639\" (UID: \"b4f9f398-a946-403d-89ff-82a803d6f639\") "
Oct 30 03:20:26.873001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.872531    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-tmp-dir" pod "b4f9f398-a946-403d-89ff-82a803d6f639" (UID: "b4f9f398-a946-403d-89ff-82a803d6f639"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:26.873001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.872642    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-home" pod "b4f9f398-a946-403d-89ff-82a803d6f639" (UID: "b4f9f398-a946-403d-89ff-82a803d6f639"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:26.873682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.873580    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:26.873682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.873605    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:26.873682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.873617    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:26.875408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.875379    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-dshm" pod "b4f9f398-a946-403d-89ff-82a803d6f639" (UID: "b4f9f398-a946-403d-89ff-82a803d6f639"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:26.875772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.875746    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f9f398-a946-403d-89ff-82a803d6f639-kube-api-access-84vj9" pod "b4f9f398-a946-403d-89ff-82a803d6f639" (UID: "b4f9f398-a946-403d-89ff-82a803d6f639"). InnerVolumeSpecName "kube-api-access-84vj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:26.876356 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.876333    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f9f398-a946-403d-89ff-82a803d6f639-tls-certs" pod "b4f9f398-a946-403d-89ff-82a803d6f639" (UID: "b4f9f398-a946-403d-89ff-82a803d6f639"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:26.900906 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.900875    4278 generic.go:356] "Generic (PLEG): container finished" podID="b4f9f398-a946-403d-89ff-82a803d6f639" containerID="2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3" exitCode=0
Oct 30 03:20:26.901071 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.900919    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm" event={"ID":"b4f9f398-a946-403d-89ff-82a803d6f639","Type":"ContainerDied","Data":"2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3"}
Oct 30 03:20:26.901071 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.900931    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"
Oct 30 03:20:26.901071 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.900965    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm" event={"ID":"b4f9f398-a946-403d-89ff-82a803d6f639","Type":"ContainerDied","Data":"4f900e9dd74387a88de0edaf0d0903b041e0f773fbad16e9b5849c4e51acaf06"}
Oct 30 03:20:26.901071 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.900988    4278 scope.go:122] "RemoveContainer" containerID="2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3"
Oct 30 03:20:26.913678 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.913655    4278 scope.go:122] "RemoveContainer" containerID="2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3"
Oct 30 03:20:26.914011 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:20:26.913991    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3\": container with ID starting with 2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3 not found: ID does not exist" containerID="2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3"
Oct 30 03:20:26.914096 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.914020    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3"} err="failed to get container status \"2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3\": rpc error: code = NotFound desc = could not find container \"2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3\": container with ID starting with 2886719f04f49db8d414853d425aa18a619aa8b300c2934892817baa678a06e3 not found: ID does not exist"
Oct 30 03:20:26.955962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.955935    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"]
Oct 30 03:20:26.960020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.959992    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/sub-test-model-ac7936-kserve-5787685d7f-nsmdm"]
Oct 30 03:20:26.974875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.974846    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f9f398-a946-403d-89ff-82a803d6f639-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:26.974875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.974871    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f9f398-a946-403d-89ff-82a803d6f639-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:26.975060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:26.974881    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84vj9\" (UniqueName: \"kubernetes.io/projected/b4f9f398-a946-403d-89ff-82a803d6f639-kube-api-access-84vj9\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:27.038174 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:27.038143    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"]
Oct 30 03:20:27.038934 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:20:27.038903    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bafe140_0707_40d7_ac96_302a8d01be97.slice/crio-8110b4bf5c0e7e5b6e13e93393ba2617d177e705679b44a70044a084dae9e8ad WatchSource:0}: Error finding container 8110b4bf5c0e7e5b6e13e93393ba2617d177e705679b44a70044a084dae9e8ad: Status 404 returned error can't find the container with id 8110b4bf5c0e7e5b6e13e93393ba2617d177e705679b44a70044a084dae9e8ad
Oct 30 03:20:27.911744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:27.911704    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7" event={"ID":"0bafe140-0707-40d7-ac96-302a8d01be97","Type":"ContainerStarted","Data":"e65657d59e32940bf6e8c950b6029c61b9d4f6721196562101155177fcb9f317"}
Oct 30 03:20:27.912251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:27.911765    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7" event={"ID":"0bafe140-0707-40d7-ac96-302a8d01be97","Type":"ContainerStarted","Data":"8110b4bf5c0e7e5b6e13e93393ba2617d177e705679b44a70044a084dae9e8ad"}
Oct 30 03:20:27.958573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:27.958508    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7" podStartSLOduration=1.958489362 podStartE2EDuration="1.958489362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:27.956931898 +0000 UTC m=+2365.881140087" watchObservedRunningTime="2026-10-30 03:20:27.958489362 +0000 UTC m=+2365.882697550"
Oct 30 03:20:28.203262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:28.203181    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f9f398-a946-403d-89ff-82a803d6f639" path="/var/lib/kubelet/pods/b4f9f398-a946-403d-89ff-82a803d6f639/volumes"
Oct 30 03:20:28.925786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:28.925720    4278 generic.go:356] "Generic (PLEG): container finished" podID="0bafe140-0707-40d7-ac96-302a8d01be97" containerID="e65657d59e32940bf6e8c950b6029c61b9d4f6721196562101155177fcb9f317" exitCode=0
Oct 30 03:20:28.926329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:28.925830    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7" event={"ID":"0bafe140-0707-40d7-ac96-302a8d01be97","Type":"ContainerDied","Data":"e65657d59e32940bf6e8c950b6029c61b9d4f6721196562101155177fcb9f317"}
Oct 30 03:20:30.098696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.098668    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:30.176921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.176874    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bafe140-0707-40d7-ac96-302a8d01be97-kube-api-access-v7wkk\" (UniqueName: \"kubernetes.io/projected/0bafe140-0707-40d7-ac96-302a8d01be97-kube-api-access-v7wkk\") pod \"0bafe140-0707-40d7-ac96-302a8d01be97\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") "
Oct 30 03:20:30.177148 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.177126    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-xks-ca\") pod \"0bafe140-0707-40d7-ac96-302a8d01be97\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") "
Oct 30 03:20:30.177255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.177222    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-openshift-ca\") pod \"0bafe140-0707-40d7-ac96-302a8d01be97\" (UID: \"0bafe140-0707-40d7-ac96-302a8d01be97\") "
Oct 30 03:20:30.177671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.177645    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-openshift-ca" pod "0bafe140-0707-40d7-ac96-302a8d01be97" (UID: "0bafe140-0707-40d7-ac96-302a8d01be97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:30.177911 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.177891    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:30.179613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.179586    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bafe140-0707-40d7-ac96-302a8d01be97-kube-api-access-v7wkk" pod "0bafe140-0707-40d7-ac96-302a8d01be97" (UID: "0bafe140-0707-40d7-ac96-302a8d01be97"). InnerVolumeSpecName "kube-api-access-v7wkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:30.180190 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.180166    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-xks-ca" pod "0bafe140-0707-40d7-ac96-302a8d01be97" (UID: "0bafe140-0707-40d7-ac96-302a8d01be97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:30.279913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.279878    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7wkk\" (UniqueName: \"kubernetes.io/projected/0bafe140-0707-40d7-ac96-302a8d01be97-kube-api-access-v7wkk\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:30.279913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.279902    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bafe140-0707-40d7-ac96-302a8d01be97-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:30.944102 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.944072    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"
Oct 30 03:20:30.944290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.944071    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7" event={"ID":"0bafe140-0707-40d7-ac96-302a8d01be97","Type":"ContainerDied","Data":"8110b4bf5c0e7e5b6e13e93393ba2617d177e705679b44a70044a084dae9e8ad"}
Oct 30 03:20:30.944290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:30.944197    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8110b4bf5c0e7e5b6e13e93393ba2617d177e705679b44a70044a084dae9e8ad"
Oct 30 03:20:37.536779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.536716    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8" podUID="283ae180-cb3b-4819-96b4-66e394ccc791" containerName="main" containerID="cri-o://5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565" gracePeriod=45
Oct 30 03:20:37.541725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.541681    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8" podUID="283ae180-cb3b-4819-96b4-66e394ccc791" containerName="main" probeResult="failure" output="Get \"https://10.134.0.187:8000/ready\": EOF"
Oct 30 03:20:37.814564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.814532    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:20:37.922404 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.922370    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-tmp-dir\") pod \"283ae180-cb3b-4819-96b4-66e394ccc791\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") "
Oct 30 03:20:37.922626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.922411    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283ae180-cb3b-4819-96b4-66e394ccc791-tls-certs\" (UniqueName: \"kubernetes.io/secret/283ae180-cb3b-4819-96b4-66e394ccc791-tls-certs\") pod \"283ae180-cb3b-4819-96b4-66e394ccc791\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") "
Oct 30 03:20:37.922626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.922476    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-home\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-home\") pod \"283ae180-cb3b-4819-96b4-66e394ccc791\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") "
Oct 30 03:20:37.922626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.922606    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-dshm\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-dshm\") pod \"283ae180-cb3b-4819-96b4-66e394ccc791\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") "
Oct 30 03:20:37.922813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.922644    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-tmp-dir" pod "283ae180-cb3b-4819-96b4-66e394ccc791" (UID: "283ae180-cb3b-4819-96b4-66e394ccc791"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.922813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.922766    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-home" pod "283ae180-cb3b-4819-96b4-66e394ccc791" (UID: "283ae180-cb3b-4819-96b4-66e394ccc791"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.922956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.922865    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283ae180-cb3b-4819-96b4-66e394ccc791-kube-api-access-lwp5f\" (UniqueName: \"kubernetes.io/projected/283ae180-cb3b-4819-96b4-66e394ccc791-kube-api-access-lwp5f\") pod \"283ae180-cb3b-4819-96b4-66e394ccc791\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") "
Oct 30 03:20:37.923001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.922981    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-model-cache\") pod \"283ae180-cb3b-4819-96b4-66e394ccc791\" (UID: \"283ae180-cb3b-4819-96b4-66e394ccc791\") "
Oct 30 03:20:37.923282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.923259    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-model-cache" pod "283ae180-cb3b-4819-96b4-66e394ccc791" (UID: "283ae180-cb3b-4819-96b4-66e394ccc791"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.923613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.923595    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.923670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.923615    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.923670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.923624    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.924961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.924939    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-dshm" pod "283ae180-cb3b-4819-96b4-66e394ccc791" (UID: "283ae180-cb3b-4819-96b4-66e394ccc791"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.925134 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.925114    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283ae180-cb3b-4819-96b4-66e394ccc791-tls-certs" pod "283ae180-cb3b-4819-96b4-66e394ccc791" (UID: "283ae180-cb3b-4819-96b4-66e394ccc791"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:37.925246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:37.925187    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283ae180-cb3b-4819-96b4-66e394ccc791-kube-api-access-lwp5f" pod "283ae180-cb3b-4819-96b4-66e394ccc791" (UID: "283ae180-cb3b-4819-96b4-66e394ccc791"). InnerVolumeSpecName "kube-api-access-lwp5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:38.025260 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.025214    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/283ae180-cb3b-4819-96b4-66e394ccc791-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:38.025260 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.025252    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwp5f\" (UniqueName: \"kubernetes.io/projected/283ae180-cb3b-4819-96b4-66e394ccc791-kube-api-access-lwp5f\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:38.025260 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.025268    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283ae180-cb3b-4819-96b4-66e394ccc791-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:38.118073 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.117970    4278 generic.go:356] "Generic (PLEG): container finished" podID="283ae180-cb3b-4819-96b4-66e394ccc791" containerID="5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565" exitCode=0
Oct 30 03:20:38.118073 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.118030    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"
Oct 30 03:20:38.118276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.118060    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8" event={"ID":"283ae180-cb3b-4819-96b4-66e394ccc791","Type":"ContainerDied","Data":"5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565"}
Oct 30 03:20:38.118276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.118112    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8" event={"ID":"283ae180-cb3b-4819-96b4-66e394ccc791","Type":"ContainerDied","Data":"a4f063eee4afe79043f8c69d41e8220b258e69bb44d0057a06d1870e27278de7"}
Oct 30 03:20:38.118276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.118132    4278 scope.go:122] "RemoveContainer" containerID="5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565"
Oct 30 03:20:38.130832 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.130809    4278 scope.go:122] "RemoveContainer" containerID="5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565"
Oct 30 03:20:38.131122 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:20:38.131101    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565\": container with ID starting with 5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565 not found: ID does not exist" containerID="5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565"
Oct 30 03:20:38.131208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.131131    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565"} err="failed to get container status \"5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565\": rpc error: code = NotFound desc = could not find container \"5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565\": container with ID starting with 5b967ca94b20535c4df99fc95a204b91240d40cb9b966079a63be89b8e768565 not found: ID does not exist"
Oct 30 03:20:38.225455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.219092    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"]
Oct 30 03:20:38.225455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.219145    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/sub-test-model-9d3552-kserve-65fd9c4c89-dgdg8"]
Oct 30 03:20:38.317060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:38.316037    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:20:40.246773 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:40.246731    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283ae180-cb3b-4819-96b4-66e394ccc791" path="/var/lib/kubelet/pods/283ae180-cb3b-4819-96b4-66e394ccc791/volumes"
Oct 30 03:20:40.776882 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:40.776838    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q" podUID="349ef879-4d46-4762-a5ec-9bb20f4ac009" containerName="main" containerID="cri-o://f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b" gracePeriod=45
Oct 30 03:20:41.082162 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.082135    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:20:41.150464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.150401    4278 generic.go:356] "Generic (PLEG): container finished" podID="349ef879-4d46-4762-a5ec-9bb20f4ac009" containerID="f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b" exitCode=0
Oct 30 03:20:41.150464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.150456    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q" event={"ID":"349ef879-4d46-4762-a5ec-9bb20f4ac009","Type":"ContainerDied","Data":"f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b"}
Oct 30 03:20:41.150724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.150474    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"
Oct 30 03:20:41.150724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.150502    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q" event={"ID":"349ef879-4d46-4762-a5ec-9bb20f4ac009","Type":"ContainerDied","Data":"153149249f183a0bcc00cb6c11ac1d706b36993d785247933f5c9ee8661d2adb"}
Oct 30 03:20:41.150724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.150525    4278 scope.go:122] "RemoveContainer" containerID="f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b"
Oct 30 03:20:41.161737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.161715    4278 scope.go:122] "RemoveContainer" containerID="f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b"
Oct 30 03:20:41.162111 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:20:41.162078    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b\": container with ID starting with f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b not found: ID does not exist" containerID="f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b"
Oct 30 03:20:41.162221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.162122    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b"} err="failed to get container status \"f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b\": rpc error: code = NotFound desc = could not find container \"f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b\": container with ID starting with f8a433bd073ad4a2783f2b9a60d2edcdd197b39059e4f61803cf68ec11b0d72b not found: ID does not exist"
Oct 30 03:20:41.211022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.210990    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-dshm\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-dshm\") pod \"349ef879-4d46-4762-a5ec-9bb20f4ac009\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") "
Oct 30 03:20:41.211212 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.211060    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-home\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-home\") pod \"349ef879-4d46-4762-a5ec-9bb20f4ac009\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") "
Oct 30 03:20:41.211212 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.211152    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349ef879-4d46-4762-a5ec-9bb20f4ac009-kube-api-access-jtswc\" (UniqueName: \"kubernetes.io/projected/349ef879-4d46-4762-a5ec-9bb20f4ac009-kube-api-access-jtswc\") pod \"349ef879-4d46-4762-a5ec-9bb20f4ac009\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") "
Oct 30 03:20:41.211337 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.211225    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-tmp-dir\") pod \"349ef879-4d46-4762-a5ec-9bb20f4ac009\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") "
Oct 30 03:20:41.211400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.211334    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-home" pod "349ef879-4d46-4762-a5ec-9bb20f4ac009" (UID: "349ef879-4d46-4762-a5ec-9bb20f4ac009"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:41.211400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.211346    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/349ef879-4d46-4762-a5ec-9bb20f4ac009-tls-certs\" (UniqueName: \"kubernetes.io/secret/349ef879-4d46-4762-a5ec-9bb20f4ac009-tls-certs\") pod \"349ef879-4d46-4762-a5ec-9bb20f4ac009\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") "
Oct 30 03:20:41.211586 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.211413    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-model-cache\") pod \"349ef879-4d46-4762-a5ec-9bb20f4ac009\" (UID: \"349ef879-4d46-4762-a5ec-9bb20f4ac009\") "
Oct 30 03:20:41.211586 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.211557    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-tmp-dir" pod "349ef879-4d46-4762-a5ec-9bb20f4ac009" (UID: "349ef879-4d46-4762-a5ec-9bb20f4ac009"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:41.211736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.211708    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-model-cache" pod "349ef879-4d46-4762-a5ec-9bb20f4ac009" (UID: "349ef879-4d46-4762-a5ec-9bb20f4ac009"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:41.212312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.212283    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:41.212492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.212315    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:41.212492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.212332    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:41.213672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.213646    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-dshm" pod "349ef879-4d46-4762-a5ec-9bb20f4ac009" (UID: "349ef879-4d46-4762-a5ec-9bb20f4ac009"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:41.213672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.213654    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349ef879-4d46-4762-a5ec-9bb20f4ac009-tls-certs" pod "349ef879-4d46-4762-a5ec-9bb20f4ac009" (UID: "349ef879-4d46-4762-a5ec-9bb20f4ac009"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:41.213802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.213722    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349ef879-4d46-4762-a5ec-9bb20f4ac009-kube-api-access-jtswc" pod "349ef879-4d46-4762-a5ec-9bb20f4ac009" (UID: "349ef879-4d46-4762-a5ec-9bb20f4ac009"). InnerVolumeSpecName "kube-api-access-jtswc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:41.314794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.314679    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtswc\" (UniqueName: \"kubernetes.io/projected/349ef879-4d46-4762-a5ec-9bb20f4ac009-kube-api-access-jtswc\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:41.314794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.314718    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349ef879-4d46-4762-a5ec-9bb20f4ac009-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:41.314794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.314729    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349ef879-4d46-4762-a5ec-9bb20f4ac009-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:20:41.504283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.504244    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"]
Oct 30 03:20:41.510380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:41.510349    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b0b40d/test-model-b0b40d-kserve-7c8544d44c-l7s4q"]
Oct 30 03:20:42.204297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.204257    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="349ef879-4d46-4762-a5ec-9bb20f4ac009" path="/var/lib/kubelet/pods/349ef879-4d46-4762-a5ec-9bb20f4ac009/volumes"
Oct 30 03:20:42.233233 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.233193    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"]
Oct 30 03:20:42.234214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234187    4278 cpu_manager.go:414] "containerMap: removing container" podUID="283ae180-cb3b-4819-96b4-66e394ccc791" containerName="main"
Oct 30 03:20:42.234214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234214    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283ae180-cb3b-4819-96b4-66e394ccc791" containerName="main"
Oct 30 03:20:42.234366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234251    4278 cpu_manager.go:414] "containerMap: removing container" podUID="b4f9f398-a946-403d-89ff-82a803d6f639" containerName="main"
Oct 30 03:20:42.234366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234258    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f9f398-a946-403d-89ff-82a803d6f639" containerName="main"
Oct 30 03:20:42.234366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234308    4278 cpu_manager.go:414] "containerMap: removing container" podUID="0bafe140-0707-40d7-ac96-302a8d01be97" containerName="revoke-keys"
Oct 30 03:20:42.234366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234316    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bafe140-0707-40d7-ac96-302a8d01be97" containerName="revoke-keys"
Oct 30 03:20:42.234366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234331    4278 cpu_manager.go:414] "containerMap: removing container" podUID="349ef879-4d46-4762-a5ec-9bb20f4ac009" containerName="main"
Oct 30 03:20:42.234366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234338    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349ef879-4d46-4762-a5ec-9bb20f4ac009" containerName="main"
Oct 30 03:20:42.234686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234597    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="283ae180-cb3b-4819-96b4-66e394ccc791" containerName="main"
Oct 30 03:20:42.234686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234620    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f9f398-a946-403d-89ff-82a803d6f639" containerName="main"
Oct 30 03:20:42.234686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234641    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bafe140-0707-40d7-ac96-302a8d01be97" containerName="revoke-keys"
Oct 30 03:20:42.234686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.234651    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="349ef879-4d46-4762-a5ec-9bb20f4ac009" containerName="main"
Oct 30 03:20:42.240214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.240186    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.257074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.256906    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"]
Oct 30 03:20:42.269645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.269614    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-6b5f19\"/\"kube-root-ca.crt\""
Oct 30 03:20:42.270147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.269966    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-6b5f19\"/\"auth-test-model-6b5f19-kserve-self-signed-certs\""
Oct 30 03:20:42.270282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.270011    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-6b5f19\"/\"default-dockercfg-pmtnh\""
Oct 30 03:20:42.270282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.270086    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-6b5f19\"/\"openshift-service-ca.crt\""
Oct 30 03:20:42.333012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.332970    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-home\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.333457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.333103    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed1592a-f1eb-457d-a101-50017d30b4d5-tls-certs\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.333457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.333173    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9z7g\" (UniqueName: \"kubernetes.io/projected/9ed1592a-f1eb-457d-a101-50017d30b4d5-kube-api-access-x9z7g\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.333457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.333243    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-model-cache\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.333457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.333361    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-dshm\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.333457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.333408    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-tmp-dir\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.435343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.435296    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-home\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.435546 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.435390    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed1592a-f1eb-457d-a101-50017d30b4d5-tls-certs\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.435546 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.435465    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9z7g\" (UniqueName: \"kubernetes.io/projected/9ed1592a-f1eb-457d-a101-50017d30b4d5-kube-api-access-x9z7g\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.435546 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.435524    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-model-cache\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.435749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.435696    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-dshm\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.435749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.435743    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-tmp-dir\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.435856 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.435804    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-home\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.436201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.436174    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-tmp-dir\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.436400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.436380    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-model-cache\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.438188 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.438161    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed1592a-f1eb-457d-a101-50017d30b4d5-tls-certs\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.438413 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.438396    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-dshm\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.457370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.457264    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9z7g\" (UniqueName: \"kubernetes.io/projected/9ed1592a-f1eb-457d-a101-50017d30b4d5-kube-api-access-x9z7g\") pod \"auth-test-model-6b5f19-kserve-7675cc6649-88sb8\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") " pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.582446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.581081    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:42.957157 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:42.957121    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"]
Oct 30 03:20:42.961529 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:20:42.961455    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ed1592a_f1eb_457d_a101_50017d30b4d5.slice/crio-000ec973026842950cc01387e3ec1fad3556276d821c33f5f96f3615a22a85fb WatchSource:0}: Error finding container 000ec973026842950cc01387e3ec1fad3556276d821c33f5f96f3615a22a85fb: Status 404 returned error can't find the container with id 000ec973026842950cc01387e3ec1fad3556276d821c33f5f96f3615a22a85fb
Oct 30 03:20:43.175918 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:43.175876    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8" event={"ID":"9ed1592a-f1eb-457d-a101-50017d30b4d5","Type":"ContainerStarted","Data":"4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59"}
Oct 30 03:20:43.176101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:43.175939    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8" event={"ID":"9ed1592a-f1eb-457d-a101-50017d30b4d5","Type":"ContainerStarted","Data":"000ec973026842950cc01387e3ec1fad3556276d821c33f5f96f3615a22a85fb"}
Oct 30 03:20:43.222746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:43.222636    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8" podStartSLOduration=1.222618628 podStartE2EDuration="1.222618628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:43.217500545 +0000 UTC m=+2381.141708735" watchObservedRunningTime="2026-10-30 03:20:43.222618628 +0000 UTC m=+2381.146826816"
Oct 30 03:20:43.587004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:43.586967    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:52.586979 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:52.586887    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:52.599870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:52.599839    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:53.273403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:53.273370    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:20:54.198023 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:20:54.197981    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:20:57.007898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.007857    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"]
Oct 30 03:20:57.014126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.014089    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.105468 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.091069    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:20:57.105468 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.091219    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:20:57.156473 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.154406    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.203380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.192965    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"]
Oct 30 03:20:57.225514 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.225474    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z7lx\" (UniqueName: \"kubernetes.io/projected/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-kube-api-access-8z7lx\") pod \"maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.257451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.240763    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.383378 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.383283    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.383626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.383409    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.383753 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.383489    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z7lx\" (UniqueName: \"kubernetes.io/projected/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-kube-api-access-8z7lx\") pod \"maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.384129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.384091    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.385953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.385929    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.404170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.404140    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z7lx\" (UniqueName: \"kubernetes.io/projected/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-kube-api-access-8z7lx\") pod \"maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.515589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.515552    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:20:57.828809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:57.828774    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"]
Oct 30 03:20:57.829852 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:20:57.829822    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee143af1_3ee2_4ff9_a6f8_cae1b09851ea.slice/crio-0f5e35a8f45336218c0eef9fa0d1b70cd1dcda6bf9f32ed09d33804bd370a30b WatchSource:0}: Error finding container 0f5e35a8f45336218c0eef9fa0d1b70cd1dcda6bf9f32ed09d33804bd370a30b: Status 404 returned error can't find the container with id 0f5e35a8f45336218c0eef9fa0d1b70cd1dcda6bf9f32ed09d33804bd370a30b
Oct 30 03:20:58.311701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:58.311655    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf" event={"ID":"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea","Type":"ContainerStarted","Data":"26f786b0baa6297088cb62d6b1e7fd279568d78c7a342185eddda6e364f85d84"}
Oct 30 03:20:58.312183 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:58.311719    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf" event={"ID":"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea","Type":"ContainerStarted","Data":"0f5e35a8f45336218c0eef9fa0d1b70cd1dcda6bf9f32ed09d33804bd370a30b"}
Oct 30 03:20:58.366899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:58.358240    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf" podStartSLOduration=2.358220744 podStartE2EDuration="2.358220744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:58.352931374 +0000 UTC m=+2396.277139562" watchObservedRunningTime="2026-10-30 03:20:58.358220744 +0000 UTC m=+2396.282428933"
Oct 30 03:20:59.321035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:59.321000    4278 generic.go:356] "Generic (PLEG): container finished" podID="ee143af1-3ee2-4ff9-a6f8-cae1b09851ea" containerID="26f786b0baa6297088cb62d6b1e7fd279568d78c7a342185eddda6e364f85d84" exitCode=0
Oct 30 03:20:59.321407 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:20:59.321080    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf" event={"ID":"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea","Type":"ContainerDied","Data":"26f786b0baa6297088cb62d6b1e7fd279568d78c7a342185eddda6e364f85d84"}
Oct 30 03:21:00.484291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:00.484267    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:21:00.526924 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:00.526894    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-openshift-ca\") pod \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") "
Oct 30 03:21:00.527099 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:00.527032    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-xks-ca\") pod \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") "
Oct 30 03:21:00.527173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:00.527102    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-kube-api-access-8z7lx\" (UniqueName: \"kubernetes.io/projected/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-kube-api-access-8z7lx\") pod \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\" (UID: \"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea\") "
Oct 30 03:21:00.527328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:00.527303    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-openshift-ca" pod "ee143af1-3ee2-4ff9-a6f8-cae1b09851ea" (UID: "ee143af1-3ee2-4ff9-a6f8-cae1b09851ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:00.527650 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:00.527628    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.529206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:00.529176    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-kube-api-access-8z7lx" pod "ee143af1-3ee2-4ff9-a6f8-cae1b09851ea" (UID: "ee143af1-3ee2-4ff9-a6f8-cae1b09851ea"). InnerVolumeSpecName "kube-api-access-8z7lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:00.529293 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:00.529177    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-xks-ca" pod "ee143af1-3ee2-4ff9-a6f8-cae1b09851ea" (UID: "ee143af1-3ee2-4ff9-a6f8-cae1b09851ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:00.628848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:00.628766    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.628848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:00.628796    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z7lx\" (UniqueName: \"kubernetes.io/projected/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea-kube-api-access-8z7lx\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:21:01.338979 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:01.338944    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"
Oct 30 03:21:01.338979 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:01.338966    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf" event={"ID":"ee143af1-3ee2-4ff9-a6f8-cae1b09851ea","Type":"ContainerDied","Data":"0f5e35a8f45336218c0eef9fa0d1b70cd1dcda6bf9f32ed09d33804bd370a30b"}
Oct 30 03:21:01.339229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:01.339030    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5e35a8f45336218c0eef9fa0d1b70cd1dcda6bf9f32ed09d33804bd370a30b"
Oct 30 03:21:05.238805 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.238771    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"]
Oct 30 03:21:05.239504 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.239463    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ee143af1-3ee2-4ff9-a6f8-cae1b09851ea" containerName="revoke-keys"
Oct 30 03:21:05.239504 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.239482    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee143af1-3ee2-4ff9-a6f8-cae1b09851ea" containerName="revoke-keys"
Oct 30 03:21:05.239644 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.239633    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee143af1-3ee2-4ff9-a6f8-cae1b09851ea" containerName="revoke-keys"
Oct 30 03:21:05.244821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.244805    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.275296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.275260    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-146bb7-serving-cert\""
Oct 30 03:21:05.286197 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.286156    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"]
Oct 30 03:21:05.297196 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.297156    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7cm7\" (UniqueName: \"kubernetes.io/projected/e873cfdf-044e-41a6-86dc-14f8c4724748-kube-api-access-w7cm7\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.297379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.297335    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-maas-api-tls\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.297618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.297600    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-metrics-certs\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.297691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.297649    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e873cfdf-044e-41a6-86dc-14f8c4724748-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.399064 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.399020    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e873cfdf-044e-41a6-86dc-14f8c4724748-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.399229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.399080    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7cm7\" (UniqueName: \"kubernetes.io/projected/e873cfdf-044e-41a6-86dc-14f8c4724748-kube-api-access-w7cm7\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.399229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.399169    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-maas-api-tls\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.399353 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.399337    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-metrics-certs\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.399815 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.399784    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e873cfdf-044e-41a6-86dc-14f8c4724748-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.401853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.401829    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-maas-api-tls\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.402061 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.402032    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-metrics-certs\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.419139 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.419108    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7cm7\" (UniqueName: \"kubernetes.io/projected/e873cfdf-044e-41a6-86dc-14f8c4724748-kube-api-access-w7cm7\") pod \"maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.560795 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.559317    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:05.882300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:05.882263    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"]
Oct 30 03:21:05.882919 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:21:05.882895    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode873cfdf_044e_41a6_86dc_14f8c4724748.slice/crio-ecd33a1f398a2cbca32b6812c5c8c09f6a503b335988c4de010fe5a2480d3e37 WatchSource:0}: Error finding container ecd33a1f398a2cbca32b6812c5c8c09f6a503b335988c4de010fe5a2480d3e37: Status 404 returned error can't find the container with id ecd33a1f398a2cbca32b6812c5c8c09f6a503b335988c4de010fe5a2480d3e37
Oct 30 03:21:06.386810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:06.386769    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6" event={"ID":"e873cfdf-044e-41a6-86dc-14f8c4724748","Type":"ContainerStarted","Data":"973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff"}
Oct 30 03:21:06.387315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:06.386849    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6" event={"ID":"e873cfdf-044e-41a6-86dc-14f8c4724748","Type":"ContainerStarted","Data":"ecd33a1f398a2cbca32b6812c5c8c09f6a503b335988c4de010fe5a2480d3e37"}
Oct 30 03:21:06.387315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:06.387104    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:06.436632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:06.436565    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6" podStartSLOduration=1.030128791 podStartE2EDuration="1.4365456s" totalImagesPullingTime="406.416809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:06.435397014 +0000 UTC m=+2404.359605215" watchObservedRunningTime="2026-10-30 03:21:06.4365456 +0000 UTC m=+2404.360753789"
Oct 30 03:21:12.400383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:12.400353    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:21:19.031711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:19.031650    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"]
Oct 30 03:21:19.043949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:19.043910    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f75cbf9-ff6rp"]
Oct 30 03:21:20.201018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:20.200982    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378fd8b1-2e02-4e04-9688-6192a71ccf21" path="/var/lib/kubelet/pods/378fd8b1-2e02-4e04-9688-6192a71ccf21/volumes"
Oct 30 03:21:24.202055 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:21:24.202004    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:21:37.035321 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:37.035280    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"]
Oct 30 03:21:37.047761 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:37.047726    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af9be9-tcddw"]
Oct 30 03:21:38.200755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:38.200720    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0509c78-4b5d-4062-814b-3a37462527ac" path="/var/lib/kubelet/pods/c0509c78-4b5d-4062-814b-3a37462527ac/volumes"
Oct 30 03:21:43.310400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:43.310357    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"]
Oct 30 03:21:43.365810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:43.365774    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0b40d-2xfrf"]
Oct 30 03:21:44.201372 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:44.201337    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee143af1-3ee2-4ff9-a6f8-cae1b09851ea" path="/var/lib/kubelet/pods/ee143af1-3ee2-4ff9-a6f8-cae1b09851ea/volumes"
Oct 30 03:21:52.165983 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.165946    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"]
Oct 30 03:21:52.171896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.171873    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.189683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.189650    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"]
Oct 30 03:21:52.240450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.240393    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdggx\" (UniqueName: \"kubernetes.io/projected/2b372c59-27aa-4b08-9820-142302cbdc86-kube-api-access-gdggx\") pod \"maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.240646 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.240600    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.240748 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.240735    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.251828 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.251801    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:21:52.251965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.251896    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:21:52.341975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.341940    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdggx\" (UniqueName: \"kubernetes.io/projected/2b372c59-27aa-4b08-9820-142302cbdc86-kube-api-access-gdggx\") pod \"maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.342739 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.342702    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.343061 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.343022    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.343807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.343784    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.345333 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.345312    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.362047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.362008    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdggx\" (UniqueName: \"kubernetes.io/projected/2b372c59-27aa-4b08-9820-142302cbdc86-kube-api-access-gdggx\") pod \"maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.489498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.489392    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:52.813115 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.813082    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"]
Oct 30 03:21:52.814002 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:21:52.813976    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b372c59_27aa_4b08_9820_142302cbdc86.slice/crio-0914f09cf805901e4f6a6fc332633e0a42f68aa8eae025f66c2e6d01d4d9b54e WatchSource:0}: Error finding container 0914f09cf805901e4f6a6fc332633e0a42f68aa8eae025f66c2e6d01d4d9b54e: Status 404 returned error can't find the container with id 0914f09cf805901e4f6a6fc332633e0a42f68aa8eae025f66c2e6d01d4d9b54e
Oct 30 03:21:52.885624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:52.885587    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk" event={"ID":"2b372c59-27aa-4b08-9820-142302cbdc86","Type":"ContainerStarted","Data":"0914f09cf805901e4f6a6fc332633e0a42f68aa8eae025f66c2e6d01d4d9b54e"}
Oct 30 03:21:53.906878 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:53.906840    4278 generic.go:356] "Generic (PLEG): container finished" podID="2b372c59-27aa-4b08-9820-142302cbdc86" containerID="a294afcba82899fc0535e97b81f440708e46a8a3c7002d6c24e0a442845fc75e" exitCode=0
Oct 30 03:21:53.908223 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:53.907209    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk" event={"ID":"2b372c59-27aa-4b08-9820-142302cbdc86","Type":"ContainerDied","Data":"a294afcba82899fc0535e97b81f440708e46a8a3c7002d6c24e0a442845fc75e"}
Oct 30 03:21:54.191413 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:21:54.191370    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:21:55.118385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.118359    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:55.213469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.213347    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b372c59-27aa-4b08-9820-142302cbdc86-kube-api-access-gdggx\" (UniqueName: \"kubernetes.io/projected/2b372c59-27aa-4b08-9820-142302cbdc86-kube-api-access-gdggx\") pod \"2b372c59-27aa-4b08-9820-142302cbdc86\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") "
Oct 30 03:21:55.213636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.213508    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-xks-ca\") pod \"2b372c59-27aa-4b08-9820-142302cbdc86\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") "
Oct 30 03:21:55.213696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.213676    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-openshift-ca\") pod \"2b372c59-27aa-4b08-9820-142302cbdc86\" (UID: \"2b372c59-27aa-4b08-9820-142302cbdc86\") "
Oct 30 03:21:55.214143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.214111    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-openshift-ca" pod "2b372c59-27aa-4b08-9820-142302cbdc86" (UID: "2b372c59-27aa-4b08-9820-142302cbdc86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:55.214379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.214357    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:21:55.215794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.215766    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-xks-ca" pod "2b372c59-27aa-4b08-9820-142302cbdc86" (UID: "2b372c59-27aa-4b08-9820-142302cbdc86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:55.215939 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.215919    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b372c59-27aa-4b08-9820-142302cbdc86-kube-api-access-gdggx" pod "2b372c59-27aa-4b08-9820-142302cbdc86" (UID: "2b372c59-27aa-4b08-9820-142302cbdc86"). InnerVolumeSpecName "kube-api-access-gdggx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:55.315589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.315539    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdggx\" (UniqueName: \"kubernetes.io/projected/2b372c59-27aa-4b08-9820-142302cbdc86-kube-api-access-gdggx\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:21:55.315589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.315581    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b372c59-27aa-4b08-9820-142302cbdc86-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:21:55.940892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.940864    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"
Oct 30 03:21:55.940892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.940857    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk" event={"ID":"2b372c59-27aa-4b08-9820-142302cbdc86","Type":"ContainerDied","Data":"0914f09cf805901e4f6a6fc332633e0a42f68aa8eae025f66c2e6d01d4d9b54e"}
Oct 30 03:21:55.941128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:21:55.940949    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0914f09cf805901e4f6a6fc332633e0a42f68aa8eae025f66c2e6d01d4d9b54e"
Oct 30 03:22:10.040414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:10.040372    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"]
Oct 30 03:22:10.053361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:10.053322    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8acd081b-9gjzs"]
Oct 30 03:22:10.200444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:10.200399    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b621ddc-d52c-41e6-a9be-63d8008f40bb" path="/var/lib/kubelet/pods/9b621ddc-d52c-41e6-a9be-63d8008f40bb/volumes"
Oct 30 03:22:11.478603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:11.478544    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:22:20.399594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.399482    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"]
Oct 30 03:22:20.401493 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.401461    4278 cpu_manager.go:414] "containerMap: removing container" podUID="2b372c59-27aa-4b08-9820-142302cbdc86" containerName="revoke-keys"
Oct 30 03:22:20.401676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.401661    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b372c59-27aa-4b08-9820-142302cbdc86" containerName="revoke-keys"
Oct 30 03:22:20.401962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.401946    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b372c59-27aa-4b08-9820-142302cbdc86" containerName="revoke-keys"
Oct 30 03:22:20.408963 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.408252    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:20.436769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.436729    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"]
Oct 30 03:22:20.505456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.459505    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:22:20.505456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.468729    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:22:20.505456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.469747    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5587d7ab-kp9ql\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:20.505456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.469924    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7f4q\" (UniqueName: \"kubernetes.io/projected/f43c028a-5967-422c-8a5b-e740c3d279cd-kube-api-access-s7f4q\") pod \"maas-api-revoke-sub-5587d7ab-kp9ql\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:20.505456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.469961    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5587d7ab-kp9ql\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:20.571592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.571556    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5587d7ab-kp9ql\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:20.571797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.571697    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7f4q\" (UniqueName: \"kubernetes.io/projected/f43c028a-5967-422c-8a5b-e740c3d279cd-kube-api-access-s7f4q\") pod \"maas-api-revoke-sub-5587d7ab-kp9ql\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:20.571797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.571724    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5587d7ab-kp9ql\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:20.572499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.572456    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5587d7ab-kp9ql\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:20.574238 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.574207    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5587d7ab-kp9ql\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:20.591750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.591721    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7f4q\" (UniqueName: \"kubernetes.io/projected/f43c028a-5967-422c-8a5b-e740c3d279cd-kube-api-access-s7f4q\") pod \"maas-api-revoke-sub-5587d7ab-kp9ql\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:20.856970 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:20.856922    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:21.161148 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:21.161113    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"]
Oct 30 03:22:21.161821 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:22:21.161794    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43c028a_5967_422c_8a5b_e740c3d279cd.slice/crio-b7d9468965567651181ae31def4d7d72b93c4a5b6de728ac247504e841e04e55 WatchSource:0}: Error finding container b7d9468965567651181ae31def4d7d72b93c4a5b6de728ac247504e841e04e55: Status 404 returned error can't find the container with id b7d9468965567651181ae31def4d7d72b93c4a5b6de728ac247504e841e04e55
Oct 30 03:22:21.293336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:21.293303    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql" event={"ID":"f43c028a-5967-422c-8a5b-e740c3d279cd","Type":"ContainerStarted","Data":"b7d9468965567651181ae31def4d7d72b93c4a5b6de728ac247504e841e04e55"}
Oct 30 03:22:22.304438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:22.304382    4278 generic.go:356] "Generic (PLEG): container finished" podID="f43c028a-5967-422c-8a5b-e740c3d279cd" containerID="a30052cb5784f695d5266e342b4acc800f952a301ba60bbf4736aed324d7c0bc" exitCode=0
Oct 30 03:22:22.304939 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:22.304484    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql" event={"ID":"f43c028a-5967-422c-8a5b-e740c3d279cd","Type":"ContainerDied","Data":"a30052cb5784f695d5266e342b4acc800f952a301ba60bbf4736aed324d7c0bc"}
Oct 30 03:22:23.191130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.191085    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"]
Oct 30 03:22:23.195996 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.195968    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.214018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.213989    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"]
Oct 30 03:22:23.338798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.338763    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.339290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.338946    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.339290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.339272    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvgz\" (UniqueName: \"kubernetes.io/projected/64d27115-9422-4d31-aa2f-e0f685f386d2-kube-api-access-dgvgz\") pod \"maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.441382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.441224    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.441382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.441335    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.441690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.441514    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvgz\" (UniqueName: \"kubernetes.io/projected/64d27115-9422-4d31-aa2f-e0f685f386d2-kube-api-access-dgvgz\") pod \"maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.442717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.442685    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.444462 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.443899    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.462838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.462809    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvgz\" (UniqueName: \"kubernetes.io/projected/64d27115-9422-4d31-aa2f-e0f685f386d2-kube-api-access-dgvgz\") pod \"maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.506816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.506785    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:23.510646 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.510616    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:23.643979 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.643931    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43c028a-5967-422c-8a5b-e740c3d279cd-kube-api-access-s7f4q\" (UniqueName: \"kubernetes.io/projected/f43c028a-5967-422c-8a5b-e740c3d279cd-kube-api-access-s7f4q\") pod \"f43c028a-5967-422c-8a5b-e740c3d279cd\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") "
Oct 30 03:22:23.644329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.644306    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-xks-ca\") pod \"f43c028a-5967-422c-8a5b-e740c3d279cd\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") "
Oct 30 03:22:23.644417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.644365    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-openshift-ca\") pod \"f43c028a-5967-422c-8a5b-e740c3d279cd\" (UID: \"f43c028a-5967-422c-8a5b-e740c3d279cd\") "
Oct 30 03:22:23.644762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.644728    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-openshift-ca" pod "f43c028a-5967-422c-8a5b-e740c3d279cd" (UID: "f43c028a-5967-422c-8a5b-e740c3d279cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:23.645100 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.645076    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:23.646809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.646779    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43c028a-5967-422c-8a5b-e740c3d279cd-kube-api-access-s7f4q" pod "f43c028a-5967-422c-8a5b-e740c3d279cd" (UID: "f43c028a-5967-422c-8a5b-e740c3d279cd"). InnerVolumeSpecName "kube-api-access-s7f4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:23.647006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.646981    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-xks-ca" pod "f43c028a-5967-422c-8a5b-e740c3d279cd" (UID: "f43c028a-5967-422c-8a5b-e740c3d279cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:22:23.747066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.746981    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43c028a-5967-422c-8a5b-e740c3d279cd-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:23.747066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.747011    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7f4q\" (UniqueName: \"kubernetes.io/projected/f43c028a-5967-422c-8a5b-e740c3d279cd-kube-api-access-s7f4q\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:23.835679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:23.835642    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"]
Oct 30 03:22:23.842291 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:22:23.842261    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d27115_9422_4d31_aa2f_e0f685f386d2.slice/crio-0e6f26c86cff64a79aa46bff1a41f2c4a3d3d2739144e2b34e6038afbd3d25e4 WatchSource:0}: Error finding container 0e6f26c86cff64a79aa46bff1a41f2c4a3d3d2739144e2b34e6038afbd3d25e4: Status 404 returned error can't find the container with id 0e6f26c86cff64a79aa46bff1a41f2c4a3d3d2739144e2b34e6038afbd3d25e4
Oct 30 03:22:24.192392 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:22:24.192355    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:22:24.323575 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:24.323531    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx" event={"ID":"64d27115-9422-4d31-aa2f-e0f685f386d2","Type":"ContainerStarted","Data":"8995c4d042ac94f449fdc49a211f3bdb18a07837110eba5fe299db12396e3f9d"}
Oct 30 03:22:24.323793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:24.323645    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx" event={"ID":"64d27115-9422-4d31-aa2f-e0f685f386d2","Type":"ContainerStarted","Data":"0e6f26c86cff64a79aa46bff1a41f2c4a3d3d2739144e2b34e6038afbd3d25e4"}
Oct 30 03:22:24.332021 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:24.331940    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"
Oct 30 03:22:24.332021 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:24.331984    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql" event={"ID":"f43c028a-5967-422c-8a5b-e740c3d279cd","Type":"ContainerDied","Data":"b7d9468965567651181ae31def4d7d72b93c4a5b6de728ac247504e841e04e55"}
Oct 30 03:22:24.332203 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:24.332045    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7d9468965567651181ae31def4d7d72b93c4a5b6de728ac247504e841e04e55"
Oct 30 03:22:24.364655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:24.364604    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx" podStartSLOduration=1.364588056 podStartE2EDuration="1.364588056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:24.363744287 +0000 UTC m=+2482.287952474" watchObservedRunningTime="2026-10-30 03:22:24.364588056 +0000 UTC m=+2482.288796244"
Oct 30 03:22:25.342920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:25.342887    4278 generic.go:356] "Generic (PLEG): container finished" podID="64d27115-9422-4d31-aa2f-e0f685f386d2" containerID="8995c4d042ac94f449fdc49a211f3bdb18a07837110eba5fe299db12396e3f9d" exitCode=0
Oct 30 03:22:25.343115 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:25.342970    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx" event={"ID":"64d27115-9422-4d31-aa2f-e0f685f386d2","Type":"ContainerDied","Data":"8995c4d042ac94f449fdc49a211f3bdb18a07837110eba5fe299db12396e3f9d"}
Oct 30 03:22:26.523444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:26.523404    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:26.695579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:26.692390    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-openshift-ca\") pod \"64d27115-9422-4d31-aa2f-e0f685f386d2\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") "
Oct 30 03:22:26.695579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:26.692930    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d27115-9422-4d31-aa2f-e0f685f386d2-kube-api-access-dgvgz\" (UniqueName: \"kubernetes.io/projected/64d27115-9422-4d31-aa2f-e0f685f386d2-kube-api-access-dgvgz\") pod \"64d27115-9422-4d31-aa2f-e0f685f386d2\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") "
Oct 30 03:22:26.695579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:26.694480    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-xks-ca\") pod \"64d27115-9422-4d31-aa2f-e0f685f386d2\" (UID: \"64d27115-9422-4d31-aa2f-e0f685f386d2\") "
Oct 30 03:22:26.700498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:26.696853    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-openshift-ca" pod "64d27115-9422-4d31-aa2f-e0f685f386d2" (UID: "64d27115-9422-4d31-aa2f-e0f685f386d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:26.700648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:26.700567    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:26.702170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:26.702142    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d27115-9422-4d31-aa2f-e0f685f386d2-kube-api-access-dgvgz" pod "64d27115-9422-4d31-aa2f-e0f685f386d2" (UID: "64d27115-9422-4d31-aa2f-e0f685f386d2"). InnerVolumeSpecName "kube-api-access-dgvgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:26.714872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:26.714842    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-xks-ca" pod "64d27115-9422-4d31-aa2f-e0f685f386d2" (UID: "64d27115-9422-4d31-aa2f-e0f685f386d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:22:26.802751 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:26.802710    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvgz\" (UniqueName: \"kubernetes.io/projected/64d27115-9422-4d31-aa2f-e0f685f386d2-kube-api-access-dgvgz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:26.802751 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:26.802752    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d27115-9422-4d31-aa2f-e0f685f386d2-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:27.373298 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:27.373268    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"
Oct 30 03:22:27.373515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:27.373295    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx" event={"ID":"64d27115-9422-4d31-aa2f-e0f685f386d2","Type":"ContainerDied","Data":"0e6f26c86cff64a79aa46bff1a41f2c4a3d3d2739144e2b34e6038afbd3d25e4"}
Oct 30 03:22:27.373515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:27.373373    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e6f26c86cff64a79aa46bff1a41f2c4a3d3d2739144e2b34e6038afbd3d25e4"
Oct 30 03:22:30.708953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.708915    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"]
Oct 30 03:22:30.709559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.709543    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f43c028a-5967-422c-8a5b-e740c3d279cd" containerName="revoke-keys"
Oct 30 03:22:30.709625 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.709560    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43c028a-5967-422c-8a5b-e740c3d279cd" containerName="revoke-keys"
Oct 30 03:22:30.709625 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.709605    4278 cpu_manager.go:414] "containerMap: removing container" podUID="64d27115-9422-4d31-aa2f-e0f685f386d2" containerName="revoke-keys"
Oct 30 03:22:30.709625 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.709611    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d27115-9422-4d31-aa2f-e0f685f386d2" containerName="revoke-keys"
Oct 30 03:22:30.709728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.709700    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d27115-9422-4d31-aa2f-e0f685f386d2" containerName="revoke-keys"
Oct 30 03:22:30.709728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.709711    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43c028a-5967-422c-8a5b-e740c3d279cd" containerName="revoke-keys"
Oct 30 03:22:30.713067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.713050    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:30.729610 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.729584    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:22:30.741579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.741557    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:22:30.746696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.746663    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"]
Oct 30 03:22:30.757075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.757048    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04334378-vp4vc\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:30.757195 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.757177    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8kz\" (UniqueName: \"kubernetes.io/projected/6136e008-a09d-498c-a9fe-275b2f9b6f4b-kube-api-access-fz8kz\") pod \"maas-api-revoke-sub-04334378-vp4vc\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:30.757239 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.757205    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04334378-vp4vc\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:30.858886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.858852    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04334378-vp4vc\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:30.859051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.858938    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8kz\" (UniqueName: \"kubernetes.io/projected/6136e008-a09d-498c-a9fe-275b2f9b6f4b-kube-api-access-fz8kz\") pod \"maas-api-revoke-sub-04334378-vp4vc\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:30.859051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.858957    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04334378-vp4vc\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:30.859693 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.859665    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04334378-vp4vc\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:30.861490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.861468    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04334378-vp4vc\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:30.879839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:30.879818    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8kz\" (UniqueName: \"kubernetes.io/projected/6136e008-a09d-498c-a9fe-275b2f9b6f4b-kube-api-access-fz8kz\") pod \"maas-api-revoke-sub-04334378-vp4vc\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:31.027192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:31.027158    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:31.331055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:31.331021    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"]
Oct 30 03:22:31.331658 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:22:31.331632    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6136e008_a09d_498c_a9fe_275b2f9b6f4b.slice/crio-381920e22c9b34e8bf586cdcd3176105ca71a37821825a3aa2266f5d1a0f81f4 WatchSource:0}: Error finding container 381920e22c9b34e8bf586cdcd3176105ca71a37821825a3aa2266f5d1a0f81f4: Status 404 returned error can't find the container with id 381920e22c9b34e8bf586cdcd3176105ca71a37821825a3aa2266f5d1a0f81f4
Oct 30 03:22:31.406459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:31.406389    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc" event={"ID":"6136e008-a09d-498c-a9fe-275b2f9b6f4b","Type":"ContainerStarted","Data":"381920e22c9b34e8bf586cdcd3176105ca71a37821825a3aa2266f5d1a0f81f4"}
Oct 30 03:22:32.416209 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:32.416174    4278 generic.go:356] "Generic (PLEG): container finished" podID="6136e008-a09d-498c-a9fe-275b2f9b6f4b" containerID="7eb87eea662ef9f96a54e1cea75edb876b40b0cb16ee4f3d285ca10d4684f0e2" exitCode=0
Oct 30 03:22:32.416567 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:32.416268    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc" event={"ID":"6136e008-a09d-498c-a9fe-275b2f9b6f4b","Type":"ContainerDied","Data":"7eb87eea662ef9f96a54e1cea75edb876b40b0cb16ee4f3d285ca10d4684f0e2"}
Oct 30 03:22:33.591640 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:33.591616    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:33.610731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:33.610704    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-openshift-ca\") pod \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") "
Oct 30 03:22:33.610880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:33.610753    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6136e008-a09d-498c-a9fe-275b2f9b6f4b-kube-api-access-fz8kz\" (UniqueName: \"kubernetes.io/projected/6136e008-a09d-498c-a9fe-275b2f9b6f4b-kube-api-access-fz8kz\") pod \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") "
Oct 30 03:22:33.610880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:33.610837    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-xks-ca\") pod \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\" (UID: \"6136e008-a09d-498c-a9fe-275b2f9b6f4b\") "
Oct 30 03:22:33.611114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:33.611092    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-openshift-ca" pod "6136e008-a09d-498c-a9fe-275b2f9b6f4b" (UID: "6136e008-a09d-498c-a9fe-275b2f9b6f4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:33.611373 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:33.611355    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:33.612866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:33.612840    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6136e008-a09d-498c-a9fe-275b2f9b6f4b-kube-api-access-fz8kz" pod "6136e008-a09d-498c-a9fe-275b2f9b6f4b" (UID: "6136e008-a09d-498c-a9fe-275b2f9b6f4b"). InnerVolumeSpecName "kube-api-access-fz8kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:33.612955 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:33.612923    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-xks-ca" pod "6136e008-a09d-498c-a9fe-275b2f9b6f4b" (UID: "6136e008-a09d-498c-a9fe-275b2f9b6f4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:22:33.713282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:33.713205    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6136e008-a09d-498c-a9fe-275b2f9b6f4b-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:33.713282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:33.713233    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz8kz\" (UniqueName: \"kubernetes.io/projected/6136e008-a09d-498c-a9fe-275b2f9b6f4b-kube-api-access-fz8kz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:34.445150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:34.445117    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"
Oct 30 03:22:34.445330 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:34.445148    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc" event={"ID":"6136e008-a09d-498c-a9fe-275b2f9b6f4b","Type":"ContainerDied","Data":"381920e22c9b34e8bf586cdcd3176105ca71a37821825a3aa2266f5d1a0f81f4"}
Oct 30 03:22:34.445330 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:34.445206    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="381920e22c9b34e8bf586cdcd3176105ca71a37821825a3aa2266f5d1a0f81f4"
Oct 30 03:22:35.029671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:35.029635    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"]
Oct 30 03:22:35.042255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:35.042226    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c653ed-7wrv5"]
Oct 30 03:22:36.201930 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:36.201896    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928e532c-1df8-4037-baa0-7773ad26a8c8" path="/var/lib/kubelet/pods/928e532c-1df8-4037-baa0-7773ad26a8c8/volumes"
Oct 30 03:22:36.967919 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:36.967875    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"]
Oct 30 03:22:36.968172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:36.968140    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6" podUID="5e12b7f9-c9e6-4aea-9b62-4031f0c71af3" containerName="maas-api" containerID="cri-o://26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361" gracePeriod=30
Oct 30 03:22:37.382507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.382120    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:22:37.526339 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.526047    4278 generic.go:356] "Generic (PLEG): container finished" podID="5e12b7f9-c9e6-4aea-9b62-4031f0c71af3" containerID="26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361" exitCode=0
Oct 30 03:22:37.526669 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.526644    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6" event={"ID":"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3","Type":"ContainerDied","Data":"26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361"}
Oct 30 03:22:37.526853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.526836    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6" event={"ID":"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3","Type":"ContainerDied","Data":"abda92064dae1848c559e759056d476d19197de8b1c4bcfc55c1722b2e11c555"}
Oct 30 03:22:37.526975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.526962    4278 scope.go:122] "RemoveContainer" containerID="26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361"
Oct 30 03:22:37.527259 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.527243    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"
Oct 30 03:22:37.573479 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.572915    4278 scope.go:122] "RemoveContainer" containerID="26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361"
Oct 30 03:22:37.573930 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:22:37.573901    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361\": container with ID starting with 26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361 not found: ID does not exist" containerID="26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361"
Oct 30 03:22:37.574063 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.573945    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361"} err="failed to get container status \"26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361\": rpc error: code = NotFound desc = could not find container \"26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361\": container with ID starting with 26197df01e6aeef370512e46af7996aa5bc9fdb71ab48ac839811f2fdbb24361 not found: ID does not exist"
Oct 30 03:22:37.586214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.586174    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-kube-api-access-lkdgt\" (UniqueName: \"kubernetes.io/projected/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-kube-api-access-lkdgt\") pod \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") "
Oct 30 03:22:37.586399 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.586296    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-maas-api-tls\") pod \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") "
Oct 30 03:22:37.586523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.586469    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-openshift-service-ca\") pod \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") "
Oct 30 03:22:37.586627 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.586606    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-metrics-certs\") pod \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\" (UID: \"5e12b7f9-c9e6-4aea-9b62-4031f0c71af3\") "
Oct 30 03:22:37.605482 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.597788    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-openshift-service-ca" pod "5e12b7f9-c9e6-4aea-9b62-4031f0c71af3" (UID: "5e12b7f9-c9e6-4aea-9b62-4031f0c71af3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:37.624445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.624394    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-maas-api-tls" pod "5e12b7f9-c9e6-4aea-9b62-4031f0c71af3" (UID: "5e12b7f9-c9e6-4aea-9b62-4031f0c71af3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:22:37.625287 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.625257    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-kube-api-access-lkdgt" pod "5e12b7f9-c9e6-4aea-9b62-4031f0c71af3" (UID: "5e12b7f9-c9e6-4aea-9b62-4031f0c71af3"). InnerVolumeSpecName "kube-api-access-lkdgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:37.627167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.627125    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-metrics-certs" pod "5e12b7f9-c9e6-4aea-9b62-4031f0c71af3" (UID: "5e12b7f9-c9e6-4aea-9b62-4031f0c71af3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:22:37.689261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.689217    4278 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-metrics-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:37.689261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.689260    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkdgt\" (UniqueName: \"kubernetes.io/projected/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-kube-api-access-lkdgt\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:37.689479 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.689276    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-maas-api-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:37.689479 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.689289    4278 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3-openshift-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:37.879092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.879062    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"]
Oct 30 03:22:37.883057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:37.883033    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4cf8f5-8b6c75556-wzss6"]
Oct 30 03:22:38.201856 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:38.201780    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e12b7f9-c9e6-4aea-9b62-4031f0c71af3" path="/var/lib/kubelet/pods/5e12b7f9-c9e6-4aea-9b62-4031f0c71af3/volumes"
Oct 30 03:22:38.202182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:38.202166    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"]
Oct 30 03:22:38.203804 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:38.203783    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4cf8f5-l9kwk"]
Oct 30 03:22:40.200294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:40.200257    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b372c59-27aa-4b08-9820-142302cbdc86" path="/var/lib/kubelet/pods/2b372c59-27aa-4b08-9820-142302cbdc86/volumes"
Oct 30 03:22:45.858170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:45.858133    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"]
Oct 30 03:22:45.858641 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:45.858413    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" podUID="6b9443ad-9188-4827-ad53-609839b11f31" containerName="istio-proxy" containerID="cri-o://8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c" gracePeriod=30
Oct 30 03:22:47.184280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.184244    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"]
Oct 30 03:22:47.185231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.185209    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6136e008-a09d-498c-a9fe-275b2f9b6f4b" containerName="revoke-keys"
Oct 30 03:22:47.185282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.185235    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6136e008-a09d-498c-a9fe-275b2f9b6f4b" containerName="revoke-keys"
Oct 30 03:22:47.185322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.185295    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5e12b7f9-c9e6-4aea-9b62-4031f0c71af3" containerName="maas-api"
Oct 30 03:22:47.185322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.185302    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e12b7f9-c9e6-4aea-9b62-4031f0c71af3" containerName="maas-api"
Oct 30 03:22:47.185499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.185484    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e12b7f9-c9e6-4aea-9b62-4031f0c71af3" containerName="maas-api"
Oct 30 03:22:47.185536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.185510    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6136e008-a09d-498c-a9fe-275b2f9b6f4b" containerName="revoke-keys"
Oct 30 03:22:47.190731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.190711    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.209380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.209357    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-4b5459-openshift-default-dockercfg-4h4xx\""
Oct 30 03:22:47.232953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.232914    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"]
Oct 30 03:22:47.240987 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.240955    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-credential-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.241177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.241079    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-podinfo\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.241177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.241115    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-data\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.241304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.241272    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istiod-ca-cert\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.241360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.241350    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-envoy\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.241412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.241380    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h929s\" (UniqueName: \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-kube-api-access-h929s\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.241510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.241495    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-token\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.241593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.241573    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-certs\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.241708 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.241688    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.376777    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istiod-ca-cert\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.376856    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-envoy\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.376882    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h929s\" (UniqueName: \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-kube-api-access-h929s\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.376947    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-token\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.377001    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-certs\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.377086    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.377179    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-credential-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.377271    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-podinfo\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.377298    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-data\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.377852    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-data\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.412652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.378222    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istiod-ca-cert\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.443866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.430777    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-credential-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.443866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.431063    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.467821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.467786    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-certs\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.486349 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.476405    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-envoy\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.536973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.512693    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h929s\" (UniqueName: \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-kube-api-access-h929s\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.536973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.517739    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-podinfo\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.536973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.524335    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-token\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.756341 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.756298    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"]
Oct 30 03:22:47.763584 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.763556    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.788045 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.788011    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"]
Oct 30 03:22:47.796466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.795387    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-948965-openshift-default-dockercfg-flldg\""
Oct 30 03:22:47.798545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.798504    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-envoy\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.798818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.798794    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db05207e-7e87-4757-a68c-e9bf54b970fc-istiod-ca-cert\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.799094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.799071    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jl2c\" (UniqueName: \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-kube-api-access-4jl2c\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.799348 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.799328    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-certs\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.799603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.799584    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-podinfo\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.799780 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.799763    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-credential-socket\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.799940 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.799925    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-socket\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.800168 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.800152    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-token\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.800363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.800347    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-data\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.805911 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.805891    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:47.902092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.902051    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-podinfo\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.902227 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.902110    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-credential-socket\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.902227 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.902149    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-socket\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.902356 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.902256    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-token\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.902437 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.902362    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-data\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.902705 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.902667    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-envoy\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.902795 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.902722    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db05207e-7e87-4757-a68c-e9bf54b970fc-istiod-ca-cert\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.902833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.902793    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jl2c\" (UniqueName: \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-kube-api-access-4jl2c\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.902870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.902860    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-credential-socket\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.902906 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.902891    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-certs\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.903319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.903291    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-certs\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.904400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.903545    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-data\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.904400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.903801    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-socket\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.904564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.904440    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db05207e-7e87-4757-a68c-e9bf54b970fc-istiod-ca-cert\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.906023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.905968    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-envoy\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.920534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.920505    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-podinfo\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.942150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.942124    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-token\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:47.942561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:47.942538    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jl2c\" (UniqueName: \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-kube-api-access-4jl2c\") pod \"e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") " pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:48.093107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.091720    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:48.245178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.245085    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:48.245797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.245243    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:48.245797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.245350    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:48.250206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.250067    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"]
Oct 30 03:22:48.569330 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:22:48.569299    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb05207e_7e87_4757_a68c_e9bf54b970fc.slice/crio-951a6a1c99df692758fedac7a51da4874438cbcdfd6ce97ed26ca704faa5c09a WatchSource:0}: Error finding container 951a6a1c99df692758fedac7a51da4874438cbcdfd6ce97ed26ca704faa5c09a: Status 404 returned error can't find the container with id 951a6a1c99df692758fedac7a51da4874438cbcdfd6ce97ed26ca704faa5c09a
Oct 30 03:22:48.569692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.569663    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"]
Oct 30 03:22:48.573549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.573498    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:48.573782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.573735    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:48.573965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.573941    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:48.631737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.631703    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" event={"ID":"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d","Type":"ContainerStarted","Data":"0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e"}
Oct 30 03:22:48.631932 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.631771    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" event={"ID":"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d","Type":"ContainerStarted","Data":"bc549a9684ccc65832a018094a211adf4a7b00af7338708c1097098d5282c571"}
Oct 30 03:22:48.633382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.633357    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" event={"ID":"db05207e-7e87-4757-a68c-e9bf54b970fc","Type":"ContainerStarted","Data":"951a6a1c99df692758fedac7a51da4874438cbcdfd6ce97ed26ca704faa5c09a"}
Oct 30 03:22:48.685793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.685721    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" podStartSLOduration=1.685702493 podStartE2EDuration="1.685702493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:48.682138154 +0000 UTC m=+2506.606346355" watchObservedRunningTime="2026-10-30 03:22:48.685702493 +0000 UTC m=+2506.609910683"
Oct 30 03:22:48.808452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:48.806531    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:49.643734 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:49.643697    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" event={"ID":"db05207e-7e87-4757-a68c-e9bf54b970fc","Type":"ContainerStarted","Data":"83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4"}
Oct 30 03:22:49.739400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:49.739329    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" podStartSLOduration=2.739306786 podStartE2EDuration="2.739306786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:49.725835276 +0000 UTC m=+2507.650043465" watchObservedRunningTime="2026-10-30 03:22:49.739306786 +0000 UTC m=+2507.663514975"
Oct 30 03:22:49.813416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:49.813382    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:50.125491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:50.125453    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:50.131945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:50.131915    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:50.651539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:50.651503    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:50.652036 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:50.651570    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:50.652615 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:50.652589    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:22:50.652842 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:50.652829    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:22:51.143209 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.143186    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:22:51.198599 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.198564    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-certs\") pod \"6b9443ad-9188-4827-ad53-609839b11f31\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") "
Oct 30 03:22:51.198824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.198786    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-envoy\") pod \"6b9443ad-9188-4827-ad53-609839b11f31\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") "
Oct 30 03:22:51.198934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.198831    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b9443ad-9188-4827-ad53-609839b11f31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9443ad-9188-4827-ad53-609839b11f31-istio-podinfo\") pod \"6b9443ad-9188-4827-ad53-609839b11f31\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") "
Oct 30 03:22:51.199315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.199245    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-kube-api-access-8d2t9\" (UniqueName: \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-kube-api-access-8d2t9\") pod \"6b9443ad-9188-4827-ad53-609839b11f31\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") "
Oct 30 03:22:51.199551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.199385    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9443ad-9188-4827-ad53-609839b11f31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9443ad-9188-4827-ad53-609839b11f31-istiod-ca-cert\") pod \"6b9443ad-9188-4827-ad53-609839b11f31\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") "
Oct 30 03:22:51.199551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.199501    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-credential-socket\") pod \"6b9443ad-9188-4827-ad53-609839b11f31\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") "
Oct 30 03:22:51.199683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.199560    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-socket\") pod \"6b9443ad-9188-4827-ad53-609839b11f31\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") "
Oct 30 03:22:51.199683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.199602    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-istio-token\" (UniqueName: \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-istio-token\") pod \"6b9443ad-9188-4827-ad53-609839b11f31\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") "
Oct 30 03:22:51.199683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.199641    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-data\") pod \"6b9443ad-9188-4827-ad53-609839b11f31\" (UID: \"6b9443ad-9188-4827-ad53-609839b11f31\") "
Oct 30 03:22:51.200520 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.198885    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-certs" pod "6b9443ad-9188-4827-ad53-609839b11f31" (UID: "6b9443ad-9188-4827-ad53-609839b11f31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:51.200670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.200650    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-data" pod "6b9443ad-9188-4827-ad53-609839b11f31" (UID: "6b9443ad-9188-4827-ad53-609839b11f31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:51.200826 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.200660    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-credential-socket" pod "6b9443ad-9188-4827-ad53-609839b11f31" (UID: "6b9443ad-9188-4827-ad53-609839b11f31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:51.200919 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.200842    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9443ad-9188-4827-ad53-609839b11f31-istiod-ca-cert" pod "6b9443ad-9188-4827-ad53-609839b11f31" (UID: "6b9443ad-9188-4827-ad53-609839b11f31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:51.200919 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.200887    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-socket" pod "6b9443ad-9188-4827-ad53-609839b11f31" (UID: "6b9443ad-9188-4827-ad53-609839b11f31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:51.202246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.202218    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-envoy" pod "6b9443ad-9188-4827-ad53-609839b11f31" (UID: "6b9443ad-9188-4827-ad53-609839b11f31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:51.204609 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.204583    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-istio-token" pod "6b9443ad-9188-4827-ad53-609839b11f31" (UID: "6b9443ad-9188-4827-ad53-609839b11f31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:51.204926 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.204898    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-kube-api-access-8d2t9" pod "6b9443ad-9188-4827-ad53-609839b11f31" (UID: "6b9443ad-9188-4827-ad53-609839b11f31"). InnerVolumeSpecName "kube-api-access-8d2t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:51.205034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.205014    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b9443ad-9188-4827-ad53-609839b11f31-istio-podinfo" pod "6b9443ad-9188-4827-ad53-609839b11f31" (UID: "6b9443ad-9188-4827-ad53-609839b11f31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:22:51.302160 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.302129    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2t9\" (UniqueName: \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-kube-api-access-8d2t9\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.302160 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.302160    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9443ad-9188-4827-ad53-609839b11f31-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.302560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.302176    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.302560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.302193    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.302560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.302205    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9443ad-9188-4827-ad53-609839b11f31-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.302560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.302217    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.302560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.302229    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.302560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.302243    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9443ad-9188-4827-ad53-609839b11f31-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.302560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.302255    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9443ad-9188-4827-ad53-609839b11f31-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.663593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.663478    4278 generic.go:356] "Generic (PLEG): container finished" podID="6b9443ad-9188-4827-ad53-609839b11f31" containerID="8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c" exitCode=0
Oct 30 03:22:51.664728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.664702    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"
Oct 30 03:22:51.673563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.671747    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" event={"ID":"6b9443ad-9188-4827-ad53-609839b11f31","Type":"ContainerDied","Data":"8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c"}
Oct 30 03:22:51.673899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.673871    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4" event={"ID":"6b9443ad-9188-4827-ad53-609839b11f31","Type":"ContainerDied","Data":"5f68f3adcd16051083d53700ffb7b0d57b5bc5ce425b92fd24408ee7afb3f1cb"}
Oct 30 03:22:51.674076 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.674060    4278 scope.go:122] "RemoveContainer" containerID="8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c"
Oct 30 03:22:51.698046 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.698020    4278 scope.go:122] "RemoveContainer" containerID="8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c"
Oct 30 03:22:51.698390 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:22:51.698368    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c\": container with ID starting with 8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c not found: ID does not exist" containerID="8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c"
Oct 30 03:22:51.698512 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.698400    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c"} err="failed to get container status \"8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c\": rpc error: code = NotFound desc = could not find container \"8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c\": container with ID starting with 8997bb6dcea5f8decf8a47092ec4e273311c05ba2f403602b8b78e702dc8484c not found: ID does not exist"
Oct 30 03:22:51.736409 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.736378    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"]
Oct 30 03:22:51.751052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:51.751017    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-f74c4"]
Oct 30 03:22:52.203500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:52.203450    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9443ad-9188-4827-ad53-609839b11f31" path="/var/lib/kubelet/pods/6b9443ad-9188-4827-ad53-609839b11f31/volumes"
Oct 30 03:22:54.031843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:54.031804    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"]
Oct 30 03:22:54.044141 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:54.044111    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21810f7d-pqjql"]
Oct 30 03:22:54.192161 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:22:54.192120    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:22:54.207900 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:54.207864    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1381e3d4-a0a1-43d1-982e-2195686b533f" path="/var/lib/kubelet/pods/1381e3d4-a0a1-43d1-982e-2195686b533f/volumes"
Oct 30 03:22:56.115806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.115756    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"]
Oct 30 03:22:56.118230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.118201    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6b9443ad-9188-4827-ad53-609839b11f31" containerName="istio-proxy"
Oct 30 03:22:56.118408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.118395    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9443ad-9188-4827-ad53-609839b11f31" containerName="istio-proxy"
Oct 30 03:22:56.118884 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.118861    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9443ad-9188-4827-ad53-609839b11f31" containerName="istio-proxy"
Oct 30 03:22:56.124190 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.124165    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:22:56.142672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.142649    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"]
Oct 30 03:22:56.149648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.149624    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-948965\""
Oct 30 03:22:56.149765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.149659    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-948965-dockercfg-pxzmw\""
Oct 30 03:22:56.236062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.219845    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd\" (UID: \"b94b1958-4b28-4e6d-8249-3dfe0b8507f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:22:56.236062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.220053    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bvl\" (UniqueName: \"kubernetes.io/projected/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-kube-api-access-46bvl\") pod \"payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd\" (UID: \"b94b1958-4b28-4e6d-8249-3dfe0b8507f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:22:56.329412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.322675    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd\" (UID: \"b94b1958-4b28-4e6d-8249-3dfe0b8507f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:22:56.340483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.329950    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bvl\" (UniqueName: \"kubernetes.io/projected/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-kube-api-access-46bvl\") pod \"payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd\" (UID: \"b94b1958-4b28-4e6d-8249-3dfe0b8507f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:22:56.360449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.327540    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd\" (UID: \"b94b1958-4b28-4e6d-8249-3dfe0b8507f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:22:56.453666 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.453545    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bvl\" (UniqueName: \"kubernetes.io/projected/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-kube-api-access-46bvl\") pod \"payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd\" (UID: \"b94b1958-4b28-4e6d-8249-3dfe0b8507f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:22:56.739213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:56.739049    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:22:57.064281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:57.064246    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"]
Oct 30 03:22:57.065017 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:22:57.064985    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94b1958_4b28_4e6d_8249_3dfe0b8507f2.slice/crio-098085428f3b97ef4e03a77d30f171b20007ebac873da2324ad9a0531eb1b14c WatchSource:0}: Error finding container 098085428f3b97ef4e03a77d30f171b20007ebac873da2324ad9a0531eb1b14c: Status 404 returned error can't find the container with id 098085428f3b97ef4e03a77d30f171b20007ebac873da2324ad9a0531eb1b14c
Oct 30 03:22:57.747201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:57.747165    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd" event={"ID":"b94b1958-4b28-4e6d-8249-3dfe0b8507f2","Type":"ContainerStarted","Data":"ea3bec75cae712f37b20d3f501721b5c025c149ea587bf5d8224252caa3ecd14"}
Oct 30 03:22:57.747722 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:57.747218    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd" event={"ID":"b94b1958-4b28-4e6d-8249-3dfe0b8507f2","Type":"ContainerStarted","Data":"098085428f3b97ef4e03a77d30f171b20007ebac873da2324ad9a0531eb1b14c"}
Oct 30 03:22:57.747722 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:57.747332    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:22:57.796734 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:22:57.794735    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd" podStartSLOduration=1.794714771 podStartE2EDuration="1.794714771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:57.794078947 +0000 UTC m=+2515.718287170" watchObservedRunningTime="2026-10-30 03:22:57.794714771 +0000 UTC m=+2515.718922962"
Oct 30 03:23:04.320771 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:04.320737    4278 scope.go:122] "RemoveContainer" containerID="5224c8962f7e448d38e1310118a0230621171e8b4c54a53e66b5b688aaecb195"
Oct 30 03:23:04.331652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:04.331632    4278 scope.go:122] "RemoveContainer" containerID="809352ad2e992ded028a64052780cfb23b6a1b9e45cfd93d458973ffce5b7cc1"
Oct 30 03:23:05.031938 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:05.031901    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"]
Oct 30 03:23:05.044668 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:05.044639    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059b73c6-v9tp6"]
Oct 30 03:23:06.261740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.261702    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd408d8-d117-45f7-8fc3-f2a1f220124f" path="/var/lib/kubelet/pods/8fd408d8-d117-45f7-8fc3-f2a1f220124f/volumes"
Oct 30 03:23:06.580179 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.580100    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"]
Oct 30 03:23:06.586310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.586287    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:06.606679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.606649    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"]
Oct 30 03:23:06.613999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.613968    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:23:06.613999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.613968    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:23:06.660338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.660300    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:06.660583 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.660361    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:06.660583 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.660497    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vclqb\" (UniqueName: \"kubernetes.io/projected/299f4f13-c7ec-413a-81c3-f244f7112a0d-kube-api-access-vclqb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:06.761519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.761489    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vclqb\" (UniqueName: \"kubernetes.io/projected/299f4f13-c7ec-413a-81c3-f244f7112a0d-kube-api-access-vclqb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:06.761738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.761722    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:06.761803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.761762    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:06.762419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.762396    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:06.764119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.764100    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:06.791535 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.791502    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vclqb\" (UniqueName: \"kubernetes.io/projected/299f4f13-c7ec-413a-81c3-f244f7112a0d-kube-api-access-vclqb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:06.900726 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:06.900652    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:07.218578 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:07.217217    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"]
Oct 30 03:23:07.221258 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:23:07.221224    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299f4f13_c7ec_413a_81c3_f244f7112a0d.slice/crio-c5889f5983cd08010f422e6a2b6829c6e60503154db1f74725123052f6af59ca WatchSource:0}: Error finding container c5889f5983cd08010f422e6a2b6829c6e60503154db1f74725123052f6af59ca: Status 404 returned error can't find the container with id c5889f5983cd08010f422e6a2b6829c6e60503154db1f74725123052f6af59ca
Oct 30 03:23:07.852878 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:07.852840    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj" event={"ID":"299f4f13-c7ec-413a-81c3-f244f7112a0d","Type":"ContainerStarted","Data":"dd397696f8ea5a67ce850d38f9947ad00e99dc32e9b65f3103dd2db828386d71"}
Oct 30 03:23:07.853289 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:07.852904    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj" event={"ID":"299f4f13-c7ec-413a-81c3-f244f7112a0d","Type":"ContainerStarted","Data":"c5889f5983cd08010f422e6a2b6829c6e60503154db1f74725123052f6af59ca"}
Oct 30 03:23:07.894340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:07.894286    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj" podStartSLOduration=1.894270755 podStartE2EDuration="1.894270755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:07.891904321 +0000 UTC m=+2525.816112533" watchObservedRunningTime="2026-10-30 03:23:07.894270755 +0000 UTC m=+2525.818478942"
Oct 30 03:23:08.805053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:08.802045    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:23:08.866006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:08.865972    4278 generic.go:356] "Generic (PLEG): container finished" podID="299f4f13-c7ec-413a-81c3-f244f7112a0d" containerID="dd397696f8ea5a67ce850d38f9947ad00e99dc32e9b65f3103dd2db828386d71" exitCode=0
Oct 30 03:23:08.866491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:08.866080    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj" event={"ID":"299f4f13-c7ec-413a-81c3-f244f7112a0d","Type":"ContainerDied","Data":"dd397696f8ea5a67ce850d38f9947ad00e99dc32e9b65f3103dd2db828386d71"}
Oct 30 03:23:10.040433 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.040396    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:10.205004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.204925    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299f4f13-c7ec-413a-81c3-f244f7112a0d-kube-api-access-vclqb\" (UniqueName: \"kubernetes.io/projected/299f4f13-c7ec-413a-81c3-f244f7112a0d-kube-api-access-vclqb\") pod \"299f4f13-c7ec-413a-81c3-f244f7112a0d\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") "
Oct 30 03:23:10.205192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.205044    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-openshift-ca\") pod \"299f4f13-c7ec-413a-81c3-f244f7112a0d\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") "
Oct 30 03:23:10.205192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.205075    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-xks-ca\") pod \"299f4f13-c7ec-413a-81c3-f244f7112a0d\" (UID: \"299f4f13-c7ec-413a-81c3-f244f7112a0d\") "
Oct 30 03:23:10.205458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.205417    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-openshift-ca" pod "299f4f13-c7ec-413a-81c3-f244f7112a0d" (UID: "299f4f13-c7ec-413a-81c3-f244f7112a0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:10.205712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.205695    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:10.207121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.207093    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299f4f13-c7ec-413a-81c3-f244f7112a0d-kube-api-access-vclqb" pod "299f4f13-c7ec-413a-81c3-f244f7112a0d" (UID: "299f4f13-c7ec-413a-81c3-f244f7112a0d"). InnerVolumeSpecName "kube-api-access-vclqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:10.207233 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.207184    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-xks-ca" pod "299f4f13-c7ec-413a-81c3-f244f7112a0d" (UID: "299f4f13-c7ec-413a-81c3-f244f7112a0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:23:10.306625 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.306586    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vclqb\" (UniqueName: \"kubernetes.io/projected/299f4f13-c7ec-413a-81c3-f244f7112a0d-kube-api-access-vclqb\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:10.306625 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.306615    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/299f4f13-c7ec-413a-81c3-f244f7112a0d-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:10.881992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.881960    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"
Oct 30 03:23:10.882177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.881999    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj" event={"ID":"299f4f13-c7ec-413a-81c3-f244f7112a0d","Type":"ContainerDied","Data":"c5889f5983cd08010f422e6a2b6829c6e60503154db1f74725123052f6af59ca"}
Oct 30 03:23:10.882177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:10.882069    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5889f5983cd08010f422e6a2b6829c6e60503154db1f74725123052f6af59ca"
Oct 30 03:23:14.039487 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:14.037911    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"]
Oct 30 03:23:14.058257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:14.051384    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21288bcc-m5b8h"]
Oct 30 03:23:14.200765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:14.200734    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cedfdb3a-4015-4ba4-b6a7-3b2558e9a183" path="/var/lib/kubelet/pods/cedfdb3a-4015-4ba4-b6a7-3b2558e9a183/volumes"
Oct 30 03:23:15.031850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:15.031814    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"]
Oct 30 03:23:15.043866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:15.043836    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8104b9fc-rztgt"]
Oct 30 03:23:16.032894 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:16.032857    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"]
Oct 30 03:23:16.045396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:16.045367    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4811b439-42fth"]
Oct 30 03:23:16.199633 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:16.199596    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71743ac3-2f89-4cf3-9f83-42f7ad51fe05" path="/var/lib/kubelet/pods/71743ac3-2f89-4cf3-9f83-42f7ad51fe05/volumes"
Oct 30 03:23:16.200117 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:16.200102    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8842ecd-e09b-455e-86dd-d31018e7ef55" path="/var/lib/kubelet/pods/f8842ecd-e09b-455e-86dd-d31018e7ef55/volumes"
Oct 30 03:23:17.104212 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:17.104174    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"]
Oct 30 03:23:22.105523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:22.105459    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7" podUID="9f865b44-0a96-4d11-a201-ed6dd3a6ade0" containerName="payload-processing" containerID="cri-o://10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960" gracePeriod=25
Oct 30 03:23:22.316053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:22.316015    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"]
Oct 30 03:23:22.338507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:22.329047    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-07e5c2-8gvnx"]
Oct 30 03:23:24.090643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:24.090597    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7" podUID="9f865b44-0a96-4d11-a201-ed6dd3a6ade0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:23:24.205992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:24.205959    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d27115-9422-4d31-aa2f-e0f685f386d2" path="/var/lib/kubelet/pods/64d27115-9422-4d31-aa2f-e0f685f386d2/volumes"
Oct 30 03:23:24.215269 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:23:24.215226    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:23:27.028999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:27.028965    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"]
Oct 30 03:23:27.040650 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:27.040622    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a42311f3-z2klm"]
Oct 30 03:23:28.200634 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:28.200599    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba" path="/var/lib/kubelet/pods/71c1f6f5-1d80-4bb3-bf49-4f2e4feac6ba/volumes"
Oct 30 03:23:34.046752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.046716    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7" podUID="9f865b44-0a96-4d11-a201-ed6dd3a6ade0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:23:34.257856 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.257825    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"]
Oct 30 03:23:34.258489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.258473    4278 cpu_manager.go:414] "containerMap: removing container" podUID="299f4f13-c7ec-413a-81c3-f244f7112a0d" containerName="revoke-keys"
Oct 30 03:23:34.258539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.258490    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299f4f13-c7ec-413a-81c3-f244f7112a0d" containerName="revoke-keys"
Oct 30 03:23:34.258608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.258592    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="299f4f13-c7ec-413a-81c3-f244f7112a0d" containerName="revoke-keys"
Oct 30 03:23:34.262201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.262184    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.280859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.280835    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"]
Oct 30 03:23:34.289737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.289154    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:23:34.289737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.289277    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:23:34.407572 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.407476    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84f57a3-a57d-4481-932f-90675de43699-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.407716 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.407569    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4d8p\" (UniqueName: \"kubernetes.io/projected/e84f57a3-a57d-4481-932f-90675de43699-kube-api-access-f4d8p\") pod \"maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.407716 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.407703    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84f57a3-a57d-4481-932f-90675de43699-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.509408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.509369    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84f57a3-a57d-4481-932f-90675de43699-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.509623 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.509441    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4d8p\" (UniqueName: \"kubernetes.io/projected/e84f57a3-a57d-4481-932f-90675de43699-kube-api-access-f4d8p\") pod \"maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.509623 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.509528    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84f57a3-a57d-4481-932f-90675de43699-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.510104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.510077    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84f57a3-a57d-4481-932f-90675de43699-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.511943 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.511922    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84f57a3-a57d-4481-932f-90675de43699-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.529529 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.529502    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4d8p\" (UniqueName: \"kubernetes.io/projected/e84f57a3-a57d-4481-932f-90675de43699-kube-api-access-f4d8p\") pod \"maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.578050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.578027    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:34.881809 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:23:34.881778    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84f57a3_a57d_4481_932f_90675de43699.slice/crio-b3b1cc7fd4a464bd4e782ce773f5bb04db77e6451af14188535b59be2d50d1e4 WatchSource:0}: Error finding container b3b1cc7fd4a464bd4e782ce773f5bb04db77e6451af14188535b59be2d50d1e4: Status 404 returned error can't find the container with id b3b1cc7fd4a464bd4e782ce773f5bb04db77e6451af14188535b59be2d50d1e4
Oct 30 03:23:34.882315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:34.882290    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"]
Oct 30 03:23:35.144263 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:35.144175    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv" event={"ID":"e84f57a3-a57d-4481-932f-90675de43699","Type":"ContainerStarted","Data":"b3b1cc7fd4a464bd4e782ce773f5bb04db77e6451af14188535b59be2d50d1e4"}
Oct 30 03:23:36.028851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:36.028813    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"]
Oct 30 03:23:36.041140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:36.041110    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ee651-xj5s2"]
Oct 30 03:23:36.154199 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:36.154110    4278 generic.go:356] "Generic (PLEG): container finished" podID="e84f57a3-a57d-4481-932f-90675de43699" containerID="3b6106ef12c0c6eec0d36b8af51e744266b6326482be298443f48d4f9f9fadd0" exitCode=0
Oct 30 03:23:36.154199 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:36.154174    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv" event={"ID":"e84f57a3-a57d-4481-932f-90675de43699","Type":"ContainerDied","Data":"3b6106ef12c0c6eec0d36b8af51e744266b6326482be298443f48d4f9f9fadd0"}
Oct 30 03:23:36.203725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:36.203691    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5cfb36-3dfa-45d8-9df2-bac35fead58e" path="/var/lib/kubelet/pods/0c5cfb36-3dfa-45d8-9df2-bac35fead58e/volumes"
Oct 30 03:23:37.062823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.062787    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"]
Oct 30 03:23:37.075373 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.075340    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51eff22b-cgr7r"]
Oct 30 03:23:37.319556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.319489    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:37.451045 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.451012    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84f57a3-a57d-4481-932f-90675de43699-kube-api-access-f4d8p\" (UniqueName: \"kubernetes.io/projected/e84f57a3-a57d-4481-932f-90675de43699-kube-api-access-f4d8p\") pod \"e84f57a3-a57d-4481-932f-90675de43699\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") "
Oct 30 03:23:37.451225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.451206    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e84f57a3-a57d-4481-932f-90675de43699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84f57a3-a57d-4481-932f-90675de43699-maas-api-xks-ca\") pod \"e84f57a3-a57d-4481-932f-90675de43699\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") "
Oct 30 03:23:37.451270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.451240    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84f57a3-a57d-4481-932f-90675de43699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84f57a3-a57d-4481-932f-90675de43699-maas-api-openshift-ca\") pod \"e84f57a3-a57d-4481-932f-90675de43699\" (UID: \"e84f57a3-a57d-4481-932f-90675de43699\") "
Oct 30 03:23:37.451681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.451652    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84f57a3-a57d-4481-932f-90675de43699-maas-api-openshift-ca" pod "e84f57a3-a57d-4481-932f-90675de43699" (UID: "e84f57a3-a57d-4481-932f-90675de43699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:37.453135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.453100    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84f57a3-a57d-4481-932f-90675de43699-kube-api-access-f4d8p" pod "e84f57a3-a57d-4481-932f-90675de43699" (UID: "e84f57a3-a57d-4481-932f-90675de43699"). InnerVolumeSpecName "kube-api-access-f4d8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:37.453247 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.453208    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84f57a3-a57d-4481-932f-90675de43699-maas-api-xks-ca" pod "e84f57a3-a57d-4481-932f-90675de43699" (UID: "e84f57a3-a57d-4481-932f-90675de43699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:23:37.553014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.552977    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4d8p\" (UniqueName: \"kubernetes.io/projected/e84f57a3-a57d-4481-932f-90675de43699-kube-api-access-f4d8p\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:37.553014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.553015    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84f57a3-a57d-4481-932f-90675de43699-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:37.553224 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:37.553031    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84f57a3-a57d-4481-932f-90675de43699-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:38.174898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:38.174870    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"
Oct 30 03:23:38.175094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:38.174899    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv" event={"ID":"e84f57a3-a57d-4481-932f-90675de43699","Type":"ContainerDied","Data":"b3b1cc7fd4a464bd4e782ce773f5bb04db77e6451af14188535b59be2d50d1e4"}
Oct 30 03:23:38.175094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:38.174985    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b1cc7fd4a464bd4e782ce773f5bb04db77e6451af14188535b59be2d50d1e4"
Oct 30 03:23:38.200987 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:38.200955    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23d2670-5f3b-4bb0-9635-795f367734dd" path="/var/lib/kubelet/pods/f23d2670-5f3b-4bb0-9635-795f367734dd/volumes"
Oct 30 03:23:42.410510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:42.410467    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:23:42.542168 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:42.542129    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-plugins-config-volume\") pod \"9f865b44-0a96-4d11-a201-ed6dd3a6ade0\" (UID: \"9f865b44-0a96-4d11-a201-ed6dd3a6ade0\") "
Oct 30 03:23:42.542369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:42.542177    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-kube-api-access-b82t7\" (UniqueName: \"kubernetes.io/projected/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-kube-api-access-b82t7\") pod \"9f865b44-0a96-4d11-a201-ed6dd3a6ade0\" (UID: \"9f865b44-0a96-4d11-a201-ed6dd3a6ade0\") "
Oct 30 03:23:42.542564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:42.542538    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-plugins-config-volume" pod "9f865b44-0a96-4d11-a201-ed6dd3a6ade0" (UID: "9f865b44-0a96-4d11-a201-ed6dd3a6ade0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:42.543011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:42.542985    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:42.544731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:42.544703    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-kube-api-access-b82t7" pod "9f865b44-0a96-4d11-a201-ed6dd3a6ade0" (UID: "9f865b44-0a96-4d11-a201-ed6dd3a6ade0"). InnerVolumeSpecName "kube-api-access-b82t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:42.662761 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:42.662625    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82t7\" (UniqueName: \"kubernetes.io/projected/9f865b44-0a96-4d11-a201-ed6dd3a6ade0-kube-api-access-b82t7\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:43.216209 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:43.216175    4278 generic.go:356] "Generic (PLEG): container finished" podID="9f865b44-0a96-4d11-a201-ed6dd3a6ade0" containerID="10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960" exitCode=0
Oct 30 03:23:43.216403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:43.216222    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"
Oct 30 03:23:43.216403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:43.216263    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7" event={"ID":"9f865b44-0a96-4d11-a201-ed6dd3a6ade0","Type":"ContainerDied","Data":"10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960"}
Oct 30 03:23:43.216403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:43.216319    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7" event={"ID":"9f865b44-0a96-4d11-a201-ed6dd3a6ade0","Type":"ContainerDied","Data":"25e1555f522be57d096fc0416201dae3d71766dd1789922c1428361b02aac90c"}
Oct 30 03:23:43.216403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:43.216348    4278 scope.go:122] "RemoveContainer" containerID="10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960"
Oct 30 03:23:43.227603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:43.227469    4278 scope.go:122] "RemoveContainer" containerID="10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960"
Oct 30 03:23:43.227807 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:23:43.227784    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960\": container with ID starting with 10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960 not found: ID does not exist" containerID="10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960"
Oct 30 03:23:43.227847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:43.227816    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960"} err="failed to get container status \"10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960\": rpc error: code = NotFound desc = could not find container \"10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960\": container with ID starting with 10309937cf5a63cfdcc5f64cf0c45d636fcc40434e9140ae9e89ed38fa549960 not found: ID does not exist"
Oct 30 03:23:43.275379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:43.275351    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"]
Oct 30 03:23:43.287118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:43.287092    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9245d1-6c69f96f98-x77n7"]
Oct 30 03:23:44.202277 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:44.202244    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f865b44-0a96-4d11-a201-ed6dd3a6ade0" path="/var/lib/kubelet/pods/9f865b44-0a96-4d11-a201-ed6dd3a6ade0/volumes"
Oct 30 03:23:49.385785 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:49.385700    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:23:53.316647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.316614    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"]
Oct 30 03:23:53.317223 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.317210    4278 cpu_manager.go:414] "containerMap: removing container" podUID="9f865b44-0a96-4d11-a201-ed6dd3a6ade0" containerName="payload-processing"
Oct 30 03:23:53.317279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.317224    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f865b44-0a96-4d11-a201-ed6dd3a6ade0" containerName="payload-processing"
Oct 30 03:23:53.317279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.317254    4278 cpu_manager.go:414] "containerMap: removing container" podUID="e84f57a3-a57d-4481-932f-90675de43699" containerName="revoke-keys"
Oct 30 03:23:53.317279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.317259    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84f57a3-a57d-4481-932f-90675de43699" containerName="revoke-keys"
Oct 30 03:23:53.317382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.317350    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84f57a3-a57d-4481-932f-90675de43699" containerName="revoke-keys"
Oct 30 03:23:53.317382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.317363    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f865b44-0a96-4d11-a201-ed6dd3a6ade0" containerName="payload-processing"
Oct 30 03:23:53.322217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.322198    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.346130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.346097    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"]
Oct 30 03:23:53.353908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.353872    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:23:53.354065 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.353943    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:23:53.383230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.383202    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7413d497-2v4gt\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.383393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.383263    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7413d497-2v4gt\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.383668 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.383640    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8jk\" (UniqueName: \"kubernetes.io/projected/7c797b9d-8a97-4d54-ab28-9da20105a44a-kube-api-access-vf8jk\") pod \"maas-api-revoke-sub-7413d497-2v4gt\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.484628 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.484598    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7413d497-2v4gt\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.484823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.484643    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7413d497-2v4gt\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.484823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.484808    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8jk\" (UniqueName: \"kubernetes.io/projected/7c797b9d-8a97-4d54-ab28-9da20105a44a-kube-api-access-vf8jk\") pod \"maas-api-revoke-sub-7413d497-2v4gt\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.485279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.485252    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7413d497-2v4gt\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.487199 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.487172    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7413d497-2v4gt\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.506276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.506250    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8jk\" (UniqueName: \"kubernetes.io/projected/7c797b9d-8a97-4d54-ab28-9da20105a44a-kube-api-access-vf8jk\") pod \"maas-api-revoke-sub-7413d497-2v4gt\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.642650 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.642556    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:53.968025 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:53.967990    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"]
Oct 30 03:23:53.973032 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:23:53.972998    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c797b9d_8a97_4d54_ab28_9da20105a44a.slice/crio-7bdb912fd984ba67f5df0c7c865ce5b93ca73eaca5eec4199c7e92195afaec6b WatchSource:0}: Error finding container 7bdb912fd984ba67f5df0c7c865ce5b93ca73eaca5eec4199c7e92195afaec6b: Status 404 returned error can't find the container with id 7bdb912fd984ba67f5df0c7c865ce5b93ca73eaca5eec4199c7e92195afaec6b
Oct 30 03:23:54.403467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:54.391040    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt" event={"ID":"7c797b9d-8a97-4d54-ab28-9da20105a44a","Type":"ContainerStarted","Data":"7bdb912fd984ba67f5df0c7c865ce5b93ca73eaca5eec4199c7e92195afaec6b"}
Oct 30 03:23:54.426409 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:23:54.426351    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:23:54.622525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:54.621316    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"]
Oct 30 03:23:54.626409 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:54.626378    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-734274-pc6mv"]
Oct 30 03:23:55.400126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:55.400090    4278 generic.go:356] "Generic (PLEG): container finished" podID="7c797b9d-8a97-4d54-ab28-9da20105a44a" containerID="72c42375f09e20915577d74efa31e4814a398f8f01e737aa49923b92990b3581" exitCode=0
Oct 30 03:23:55.400298 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:55.400173    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt" event={"ID":"7c797b9d-8a97-4d54-ab28-9da20105a44a","Type":"ContainerDied","Data":"72c42375f09e20915577d74efa31e4814a398f8f01e737aa49923b92990b3581"}
Oct 30 03:23:56.200503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.200471    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84f57a3-a57d-4481-932f-90675de43699" path="/var/lib/kubelet/pods/e84f57a3-a57d-4481-932f-90675de43699/volumes"
Oct 30 03:23:56.563953 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.563927    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:56.593631 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.593604    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-xks-ca\") pod \"7c797b9d-8a97-4d54-ab28-9da20105a44a\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") "
Oct 30 03:23:56.593779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.593692    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c797b9d-8a97-4d54-ab28-9da20105a44a-kube-api-access-vf8jk\" (UniqueName: \"kubernetes.io/projected/7c797b9d-8a97-4d54-ab28-9da20105a44a-kube-api-access-vf8jk\") pod \"7c797b9d-8a97-4d54-ab28-9da20105a44a\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") "
Oct 30 03:23:56.593779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.593754    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-openshift-ca\") pod \"7c797b9d-8a97-4d54-ab28-9da20105a44a\" (UID: \"7c797b9d-8a97-4d54-ab28-9da20105a44a\") "
Oct 30 03:23:56.594110 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.594085    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-openshift-ca" pod "7c797b9d-8a97-4d54-ab28-9da20105a44a" (UID: "7c797b9d-8a97-4d54-ab28-9da20105a44a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:56.594234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.594214    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:56.595830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.595801    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c797b9d-8a97-4d54-ab28-9da20105a44a-kube-api-access-vf8jk" pod "7c797b9d-8a97-4d54-ab28-9da20105a44a" (UID: "7c797b9d-8a97-4d54-ab28-9da20105a44a"). InnerVolumeSpecName "kube-api-access-vf8jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:56.595964 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.595891    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-xks-ca" pod "7c797b9d-8a97-4d54-ab28-9da20105a44a" (UID: "7c797b9d-8a97-4d54-ab28-9da20105a44a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:23:56.695866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.695833    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c797b9d-8a97-4d54-ab28-9da20105a44a-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:56.695866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:56.695862    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8jk\" (UniqueName: \"kubernetes.io/projected/7c797b9d-8a97-4d54-ab28-9da20105a44a-kube-api-access-vf8jk\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:23:57.417370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:57.417340    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"
Oct 30 03:23:57.417871 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:57.417384    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt" event={"ID":"7c797b9d-8a97-4d54-ab28-9da20105a44a","Type":"ContainerDied","Data":"7bdb912fd984ba67f5df0c7c865ce5b93ca73eaca5eec4199c7e92195afaec6b"}
Oct 30 03:23:57.417871 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:57.417452    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bdb912fd984ba67f5df0c7c865ce5b93ca73eaca5eec4199c7e92195afaec6b"
Oct 30 03:23:59.949488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:59.946108    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"]
Oct 30 03:23:59.949488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:23:59.946397    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h" podUID="a0a3fdf8-344b-4be7-a56b-6edd1489b992" containerName="maas-api" containerID="cri-o://30f49a31badc8d413c0e1086bf17035a0af5fc4835c87572d667606cbc609d60" gracePeriod=30
Oct 30 03:24:00.550459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.535246    4278 generic.go:356] "Generic (PLEG): container finished" podID="a0a3fdf8-344b-4be7-a56b-6edd1489b992" containerID="30f49a31badc8d413c0e1086bf17035a0af5fc4835c87572d667606cbc609d60" exitCode=0
Oct 30 03:24:00.550459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.535347    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h" event={"ID":"a0a3fdf8-344b-4be7-a56b-6edd1489b992","Type":"ContainerDied","Data":"30f49a31badc8d413c0e1086bf17035a0af5fc4835c87572d667606cbc609d60"}
Oct 30 03:24:00.550459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.535393    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h" event={"ID":"a0a3fdf8-344b-4be7-a56b-6edd1489b992","Type":"ContainerDied","Data":"b877b8dfe6aeb6f0a4e28e9f03745e79432266ebefef72963ec5f2a67c565c07"}
Oct 30 03:24:00.550459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.535415    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b877b8dfe6aeb6f0a4e28e9f03745e79432266ebefef72963ec5f2a67c565c07"
Oct 30 03:24:00.550459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.539127    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:24:00.712731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.712646    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a3fdf8-344b-4be7-a56b-6edd1489b992-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a3fdf8-344b-4be7-a56b-6edd1489b992-openshift-service-ca\") pod \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") "
Oct 30 03:24:00.712731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.712699    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-maas-api-tls\") pod \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") "
Oct 30 03:24:00.712937 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.712732    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-metrics-certs\") pod \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") "
Oct 30 03:24:00.712937 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.712822    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a3fdf8-344b-4be7-a56b-6edd1489b992-kube-api-access-c4t2r\" (UniqueName: \"kubernetes.io/projected/a0a3fdf8-344b-4be7-a56b-6edd1489b992-kube-api-access-c4t2r\") pod \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\" (UID: \"a0a3fdf8-344b-4be7-a56b-6edd1489b992\") "
Oct 30 03:24:00.713049 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.713026    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a3fdf8-344b-4be7-a56b-6edd1489b992-openshift-service-ca" pod "a0a3fdf8-344b-4be7-a56b-6edd1489b992" (UID: "a0a3fdf8-344b-4be7-a56b-6edd1489b992"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:00.713606 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.713580    4278 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a3fdf8-344b-4be7-a56b-6edd1489b992-openshift-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:00.714968 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.714942    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-maas-api-tls" pod "a0a3fdf8-344b-4be7-a56b-6edd1489b992" (UID: "a0a3fdf8-344b-4be7-a56b-6edd1489b992"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:00.715095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.714994    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a3fdf8-344b-4be7-a56b-6edd1489b992-kube-api-access-c4t2r" pod "a0a3fdf8-344b-4be7-a56b-6edd1489b992" (UID: "a0a3fdf8-344b-4be7-a56b-6edd1489b992"). InnerVolumeSpecName "kube-api-access-c4t2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:00.715095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.715008    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-metrics-certs" pod "a0a3fdf8-344b-4be7-a56b-6edd1489b992" (UID: "a0a3fdf8-344b-4be7-a56b-6edd1489b992"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:00.814738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.814697    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-maas-api-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:00.814738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.814735    4278 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0a3fdf8-344b-4be7-a56b-6edd1489b992-metrics-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:00.814738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.814746    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4t2r\" (UniqueName: \"kubernetes.io/projected/a0a3fdf8-344b-4be7-a56b-6edd1489b992-kube-api-access-c4t2r\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:00.839721 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.839687    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"]
Oct 30 03:24:00.843129 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:00.843108    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9245d1-h9bnj"]
Oct 30 03:24:01.543193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:01.543157    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"
Oct 30 03:24:01.603816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:01.603784    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"]
Oct 30 03:24:01.616621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:01.616592    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9245d1-56c94fdff4-g842h"]
Oct 30 03:24:02.200285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.200249    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299f4f13-c7ec-413a-81c3-f244f7112a0d" path="/var/lib/kubelet/pods/299f4f13-c7ec-413a-81c3-f244f7112a0d/volumes"
Oct 30 03:24:02.200806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.200788    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a3fdf8-344b-4be7-a56b-6edd1489b992" path="/var/lib/kubelet/pods/a0a3fdf8-344b-4be7-a56b-6edd1489b992/volumes"
Oct 30 03:24:02.950658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.950620    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"]
Oct 30 03:24:02.951220 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.951207    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a0a3fdf8-344b-4be7-a56b-6edd1489b992" containerName="maas-api"
Oct 30 03:24:02.951271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.951222    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a3fdf8-344b-4be7-a56b-6edd1489b992" containerName="maas-api"
Oct 30 03:24:02.951271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.951247    4278 cpu_manager.go:414] "containerMap: removing container" podUID="7c797b9d-8a97-4d54-ab28-9da20105a44a" containerName="revoke-keys"
Oct 30 03:24:02.951271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.951253    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c797b9d-8a97-4d54-ab28-9da20105a44a" containerName="revoke-keys"
Oct 30 03:24:02.951373 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.951349    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a3fdf8-344b-4be7-a56b-6edd1489b992" containerName="maas-api"
Oct 30 03:24:02.951373 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.951361    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c797b9d-8a97-4d54-ab28-9da20105a44a" containerName="revoke-keys"
Oct 30 03:24:02.955063 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.955043    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:02.978719 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.978686    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"]
Oct 30 03:24:02.984814 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.984771    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:24:02.984986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:02.984824    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:24:03.053419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.053373    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:03.053613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.053569    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5cpm\" (UniqueName: \"kubernetes.io/projected/307eda7e-9b6a-4ef2-9786-2b872dce22b1-kube-api-access-s5cpm\") pod \"maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:03.053800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.053774    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:03.154864 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.154831    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:03.155038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.154924    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5cpm\" (UniqueName: \"kubernetes.io/projected/307eda7e-9b6a-4ef2-9786-2b872dce22b1-kube-api-access-s5cpm\") pod \"maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:03.155093 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.155054    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:03.155595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.155569    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:03.157393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.157365    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:03.176687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.176661    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5cpm\" (UniqueName: \"kubernetes.io/projected/307eda7e-9b6a-4ef2-9786-2b872dce22b1-kube-api-access-s5cpm\") pod \"maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:03.268862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.268823    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:03.589616 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:24:03.589586    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307eda7e_9b6a_4ef2_9786_2b872dce22b1.slice/crio-d8cff89925c9d23aad1be83b16ca32d5b06855310883f2289577ecd2c3f37d92 WatchSource:0}: Error finding container d8cff89925c9d23aad1be83b16ca32d5b06855310883f2289577ecd2c3f37d92: Status 404 returned error can't find the container with id d8cff89925c9d23aad1be83b16ca32d5b06855310883f2289577ecd2c3f37d92
Oct 30 03:24:03.590314 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.590286    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"]
Oct 30 03:24:03.716691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.716657    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"]
Oct 30 03:24:03.722028 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.722007    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:03.742656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.742630    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"]
Oct 30 03:24:03.870396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.870314    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2r98\" (UniqueName: \"kubernetes.io/projected/f1c87bcc-494c-42cf-ace8-3c4692d53c47-kube-api-access-c2r98\") pod \"maas-api-revoke-sub-33ffa757-sh7bn\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:03.870580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.870533    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33ffa757-sh7bn\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:03.870640 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.870606    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33ffa757-sh7bn\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:03.972838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.972795    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33ffa757-sh7bn\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:03.973547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.972852    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33ffa757-sh7bn\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:03.973547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.973108    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2r98\" (UniqueName: \"kubernetes.io/projected/f1c87bcc-494c-42cf-ace8-3c4692d53c47-kube-api-access-c2r98\") pod \"maas-api-revoke-sub-33ffa757-sh7bn\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:03.975021 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.974992    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33ffa757-sh7bn\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:03.976848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.976818    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33ffa757-sh7bn\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:03.993560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:03.993532    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2r98\" (UniqueName: \"kubernetes.io/projected/f1c87bcc-494c-42cf-ace8-3c4692d53c47-kube-api-access-c2r98\") pod \"maas-api-revoke-sub-33ffa757-sh7bn\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:04.037188 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.037162    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:04.357477 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.357453    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"]
Oct 30 03:24:04.358409 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:24:04.358388    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c87bcc_494c_42cf_ace8_3c4692d53c47.slice/crio-bc7f4a7a4c02cc366e54c0a01ddbafbd581fb232f092455023566d0ca424099c WatchSource:0}: Error finding container bc7f4a7a4c02cc366e54c0a01ddbafbd581fb232f092455023566d0ca424099c: Status 404 returned error can't find the container with id bc7f4a7a4c02cc366e54c0a01ddbafbd581fb232f092455023566d0ca424099c
Oct 30 03:24:04.381307 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.381282    4278 scope.go:122] "RemoveContainer" containerID="5976fc20e905778e87789241174766b99d1cf081394ba8779fe4ff7a973c55f0"
Oct 30 03:24:04.394189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.394160    4278 scope.go:122] "RemoveContainer" containerID="37201fd8ed365b12b30f6251dbde89078a07872c5d8660a4acd9fc8d3b95eccb"
Oct 30 03:24:04.410403 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.410382    4278 scope.go:122] "RemoveContainer" containerID="4c9e6f26381fa5e35ada310c1ae99cbd905a1ae7300d8e7b053094c1da0c4a13"
Oct 30 03:24:04.419393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.419365    4278 scope.go:122] "RemoveContainer" containerID="df994648d9dac2c3304e62deeb2db96daff94d4808f5c1cb33d0a5702904ad88"
Oct 30 03:24:04.429113 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.429095    4278 scope.go:122] "RemoveContainer" containerID="401ffcfeea5c34587700440ad70a74875ecc1af409f388cee866a692effba10d"
Oct 30 03:24:04.437968 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.437951    4278 scope.go:122] "RemoveContainer" containerID="e282d00b3699da571396e09df90e22e096d2c4cf162f037676159862d0b5635b"
Oct 30 03:24:04.447599 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.447581    4278 scope.go:122] "RemoveContainer" containerID="2efed3400f59b8e12e231dcc4df1f11146131fdd0b842b57091b0b394a6552a9"
Oct 30 03:24:04.456771 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.456756    4278 scope.go:122] "RemoveContainer" containerID="30f49a31badc8d413c0e1086bf17035a0af5fc4835c87572d667606cbc609d60"
Oct 30 03:24:04.595997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.595959    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz" event={"ID":"307eda7e-9b6a-4ef2-9786-2b872dce22b1","Type":"ContainerStarted","Data":"2f6ab60a35d24a1d2ab16bb186e05cbce04421b938cc4b2b45375a0c7e447f7d"}
Oct 30 03:24:04.596166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.596013    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz" event={"ID":"307eda7e-9b6a-4ef2-9786-2b872dce22b1","Type":"ContainerStarted","Data":"d8cff89925c9d23aad1be83b16ca32d5b06855310883f2289577ecd2c3f37d92"}
Oct 30 03:24:04.597574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.597550    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn" event={"ID":"f1c87bcc-494c-42cf-ace8-3c4692d53c47","Type":"ContainerStarted","Data":"bc7f4a7a4c02cc366e54c0a01ddbafbd581fb232f092455023566d0ca424099c"}
Oct 30 03:24:04.637413 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:04.637323    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz" podStartSLOduration=2.637308633 podStartE2EDuration="2.637308633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:04.635964764 +0000 UTC m=+2582.560172949" watchObservedRunningTime="2026-10-30 03:24:04.637308633 +0000 UTC m=+2582.561516821"
Oct 30 03:24:05.606331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:05.606251    4278 generic.go:356] "Generic (PLEG): container finished" podID="307eda7e-9b6a-4ef2-9786-2b872dce22b1" containerID="2f6ab60a35d24a1d2ab16bb186e05cbce04421b938cc4b2b45375a0c7e447f7d" exitCode=0
Oct 30 03:24:05.607070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:05.606339    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz" event={"ID":"307eda7e-9b6a-4ef2-9786-2b872dce22b1","Type":"ContainerDied","Data":"2f6ab60a35d24a1d2ab16bb186e05cbce04421b938cc4b2b45375a0c7e447f7d"}
Oct 30 03:24:05.608002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:05.607983    4278 generic.go:356] "Generic (PLEG): container finished" podID="f1c87bcc-494c-42cf-ace8-3c4692d53c47" containerID="cd664a5b34ba29ef3a50c114f6ae14cba0d66f78a678dccf093c5f9e666c763f" exitCode=0
Oct 30 03:24:05.608109 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:05.608013    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn" event={"ID":"f1c87bcc-494c-42cf-ace8-3c4692d53c47","Type":"ContainerDied","Data":"cd664a5b34ba29ef3a50c114f6ae14cba0d66f78a678dccf093c5f9e666c763f"}
Oct 30 03:24:06.794919 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.794891    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:06.816766 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.816736    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-xks-ca\") pod \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") "
Oct 30 03:24:06.817044 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.817019    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-openshift-ca\") pod \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") "
Oct 30 03:24:06.817140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.817121    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c87bcc-494c-42cf-ace8-3c4692d53c47-kube-api-access-c2r98\" (UniqueName: \"kubernetes.io/projected/f1c87bcc-494c-42cf-ace8-3c4692d53c47-kube-api-access-c2r98\") pod \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\" (UID: \"f1c87bcc-494c-42cf-ace8-3c4692d53c47\") "
Oct 30 03:24:06.817354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.817325    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-openshift-ca" pod "f1c87bcc-494c-42cf-ace8-3c4692d53c47" (UID: "f1c87bcc-494c-42cf-ace8-3c4692d53c47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:06.817756 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.817735    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:06.818853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.818825    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-xks-ca" pod "f1c87bcc-494c-42cf-ace8-3c4692d53c47" (UID: "f1c87bcc-494c-42cf-ace8-3c4692d53c47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:06.819088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.819069    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c87bcc-494c-42cf-ace8-3c4692d53c47-kube-api-access-c2r98" pod "f1c87bcc-494c-42cf-ace8-3c4692d53c47" (UID: "f1c87bcc-494c-42cf-ace8-3c4692d53c47"). InnerVolumeSpecName "kube-api-access-c2r98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:06.842040 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.842013    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:06.918643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.918558    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307eda7e-9b6a-4ef2-9786-2b872dce22b1-kube-api-access-s5cpm\" (UniqueName: \"kubernetes.io/projected/307eda7e-9b6a-4ef2-9786-2b872dce22b1-kube-api-access-s5cpm\") pod \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") "
Oct 30 03:24:06.918643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.918621    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-openshift-ca\") pod \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") "
Oct 30 03:24:06.918879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.918650    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-xks-ca\") pod \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\" (UID: \"307eda7e-9b6a-4ef2-9786-2b872dce22b1\") "
Oct 30 03:24:06.919157 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.919025    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-openshift-ca" pod "307eda7e-9b6a-4ef2-9786-2b872dce22b1" (UID: "307eda7e-9b6a-4ef2-9786-2b872dce22b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:06.919551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.919526    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2r98\" (UniqueName: \"kubernetes.io/projected/f1c87bcc-494c-42cf-ace8-3c4692d53c47-kube-api-access-c2r98\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:06.919663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.919557    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:06.919663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.919577    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c87bcc-494c-42cf-ace8-3c4692d53c47-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:06.921012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.920978    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-xks-ca" pod "307eda7e-9b6a-4ef2-9786-2b872dce22b1" (UID: "307eda7e-9b6a-4ef2-9786-2b872dce22b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:06.921012 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:06.920993    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307eda7e-9b6a-4ef2-9786-2b872dce22b1-kube-api-access-s5cpm" pod "307eda7e-9b6a-4ef2-9786-2b872dce22b1" (UID: "307eda7e-9b6a-4ef2-9786-2b872dce22b1"). InnerVolumeSpecName "kube-api-access-s5cpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:07.021569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:07.021531    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5cpm\" (UniqueName: \"kubernetes.io/projected/307eda7e-9b6a-4ef2-9786-2b872dce22b1-kube-api-access-s5cpm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:07.021569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:07.021563    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307eda7e-9b6a-4ef2-9786-2b872dce22b1-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:07.624207 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:07.624178    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"
Oct 30 03:24:07.624419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:07.624200    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn" event={"ID":"f1c87bcc-494c-42cf-ace8-3c4692d53c47","Type":"ContainerDied","Data":"bc7f4a7a4c02cc366e54c0a01ddbafbd581fb232f092455023566d0ca424099c"}
Oct 30 03:24:07.624419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:07.624273    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc7f4a7a4c02cc366e54c0a01ddbafbd581fb232f092455023566d0ca424099c"
Oct 30 03:24:07.626159 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:07.626124    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz" event={"ID":"307eda7e-9b6a-4ef2-9786-2b872dce22b1","Type":"ContainerDied","Data":"d8cff89925c9d23aad1be83b16ca32d5b06855310883f2289577ecd2c3f37d92"}
Oct 30 03:24:07.626305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:07.626167    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8cff89925c9d23aad1be83b16ca32d5b06855310883f2289577ecd2c3f37d92"
Oct 30 03:24:07.626305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:07.626136    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"
Oct 30 03:24:07.738906 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:07.738868    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"]
Oct 30 03:24:08.406564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:08.406533    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"]
Oct 30 03:24:08.406942 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:08.406743    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy" containerID="cri-o://77b5b5a1aacf1ff2167ed1a2d3c3d2965c9e23debc9d91ce56d2274304e3766d" gracePeriod=30
Oct 30 03:24:08.423731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:08.423690    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"]
Oct 30 03:24:08.424214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:08.424164    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy" containerID="cri-o://d9388cd3431f23ba24d2037dab3aabb6e370f7a449b01bfb8d3f56e4ecd3bea5" gracePeriod=30
Oct 30 03:24:08.436568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:08.436531    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"]
Oct 30 03:24:08.436809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:08.436779    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerName="istio-proxy" containerID="cri-o://5fbfebaa37d99c74631dcc3cd48527b9cb7b4ed64a8f05e146e821e94bbb766f" gracePeriod=30
Oct 30 03:24:09.016266 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.016227    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"]
Oct 30 03:24:09.016863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.016848    4278 cpu_manager.go:414] "containerMap: removing container" podUID="307eda7e-9b6a-4ef2-9786-2b872dce22b1" containerName="revoke-keys"
Oct 30 03:24:09.016934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.016865    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307eda7e-9b6a-4ef2-9786-2b872dce22b1" containerName="revoke-keys"
Oct 30 03:24:09.016934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.016885    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f1c87bcc-494c-42cf-ace8-3c4692d53c47" containerName="revoke-keys"
Oct 30 03:24:09.016934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.016892    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c87bcc-494c-42cf-ace8-3c4692d53c47" containerName="revoke-keys"
Oct 30 03:24:09.017033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.016999    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c87bcc-494c-42cf-ace8-3c4692d53c47" containerName="revoke-keys"
Oct 30 03:24:09.017033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.017016    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="307eda7e-9b6a-4ef2-9786-2b872dce22b1" containerName="revoke-keys"
Oct 30 03:24:09.020806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.020785    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.044724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.044694    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"]
Oct 30 03:24:09.049954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.049920    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:24:09.050833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.050813    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:24:09.053362 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.053337    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcqhs\" (UniqueName: \"kubernetes.io/projected/cd1579bf-0969-4f13-9850-7f56e08c1cc2-kube-api-access-gcqhs\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.053586 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.053564    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.053676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.053607    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.157353 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.157313    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.157747 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.157717    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.158150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.158127    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcqhs\" (UniqueName: \"kubernetes.io/projected/cd1579bf-0969-4f13-9850-7f56e08c1cc2-kube-api-access-gcqhs\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.169087 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.168899    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.171882 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.171855    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.179083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.179055    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcqhs\" (UniqueName: \"kubernetes.io/projected/cd1579bf-0969-4f13-9850-7f56e08c1cc2-kube-api-access-gcqhs\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.337729 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.337636    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:09.643090 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.643061    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"]
Oct 30 03:24:09.643650 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:24:09.643625    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1579bf_0969_4f13_9850_7f56e08c1cc2.slice/crio-63e19314405e07329b92216a8a79be33a527cf2f76c64705f8ace27b31b7bcdb WatchSource:0}: Error finding container 63e19314405e07329b92216a8a79be33a527cf2f76c64705f8ace27b31b7bcdb: Status 404 returned error can't find the container with id 63e19314405e07329b92216a8a79be33a527cf2f76c64705f8ace27b31b7bcdb
Oct 30 03:24:09.662488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:09.662463    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr" event={"ID":"cd1579bf-0969-4f13-9850-7f56e08c1cc2","Type":"ContainerStarted","Data":"63e19314405e07329b92216a8a79be33a527cf2f76c64705f8ace27b31b7bcdb"}
Oct 30 03:24:10.680397 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:10.680354    4278 generic.go:356] "Generic (PLEG): container finished" podID="cd1579bf-0969-4f13-9850-7f56e08c1cc2" containerID="11961fbdb924d1c3fa8217f705b063fa20548d219d3924c2d653f8f5af3160aa" exitCode=0
Oct 30 03:24:10.680794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:10.680449    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr" event={"ID":"cd1579bf-0969-4f13-9850-7f56e08c1cc2","Type":"ContainerDied","Data":"11961fbdb924d1c3fa8217f705b063fa20548d219d3924c2d653f8f5af3160aa"}
Oct 30 03:24:11.841838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:11.841811    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:11.906338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:11.906296    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-xks-ca\") pod \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") "
Oct 30 03:24:11.906596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:11.906574    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1579bf-0969-4f13-9850-7f56e08c1cc2-kube-api-access-gcqhs\" (UniqueName: \"kubernetes.io/projected/cd1579bf-0969-4f13-9850-7f56e08c1cc2-kube-api-access-gcqhs\") pod \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") "
Oct 30 03:24:11.906678 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:11.906662    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-openshift-ca\") pod \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\" (UID: \"cd1579bf-0969-4f13-9850-7f56e08c1cc2\") "
Oct 30 03:24:11.907158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:11.907096    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-openshift-ca" pod "cd1579bf-0969-4f13-9850-7f56e08c1cc2" (UID: "cd1579bf-0969-4f13-9850-7f56e08c1cc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:11.907444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:11.907404    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:11.908689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:11.908657    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1579bf-0969-4f13-9850-7f56e08c1cc2-kube-api-access-gcqhs" pod "cd1579bf-0969-4f13-9850-7f56e08c1cc2" (UID: "cd1579bf-0969-4f13-9850-7f56e08c1cc2"). InnerVolumeSpecName "kube-api-access-gcqhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:11.908983 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:11.908958    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-xks-ca" pod "cd1579bf-0969-4f13-9850-7f56e08c1cc2" (UID: "cd1579bf-0969-4f13-9850-7f56e08c1cc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:12.008988 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:12.008958    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1579bf-0969-4f13-9850-7f56e08c1cc2-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:12.008988 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:12.008986    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcqhs\" (UniqueName: \"kubernetes.io/projected/cd1579bf-0969-4f13-9850-7f56e08c1cc2-kube-api-access-gcqhs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:12.697208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:12.697182    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"
Oct 30 03:24:12.697380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:12.697270    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr" event={"ID":"cd1579bf-0969-4f13-9850-7f56e08c1cc2","Type":"ContainerDied","Data":"63e19314405e07329b92216a8a79be33a527cf2f76c64705f8ace27b31b7bcdb"}
Oct 30 03:24:12.697380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:12.697311    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63e19314405e07329b92216a8a79be33a527cf2f76c64705f8ace27b31b7bcdb"
Oct 30 03:24:13.712558 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.712487    4278 generic.go:356] "Generic (PLEG): container finished" podID="9e31519d-e465-4414-b22b-554f662ae87f" containerID="77b5b5a1aacf1ff2167ed1a2d3c3d2965c9e23debc9d91ce56d2274304e3766d" exitCode=0
Oct 30 03:24:13.713023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.712667    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" event={"ID":"9e31519d-e465-4414-b22b-554f662ae87f","Type":"ContainerDied","Data":"77b5b5a1aacf1ff2167ed1a2d3c3d2965c9e23debc9d91ce56d2274304e3766d"}
Oct 30 03:24:13.715390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.715360    4278 generic.go:356] "Generic (PLEG): container finished" podID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerID="5fbfebaa37d99c74631dcc3cd48527b9cb7b4ed64a8f05e146e821e94bbb766f" exitCode=0
Oct 30 03:24:13.715615 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.715596    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" event={"ID":"137707a1-f8b6-4f13-8eb3-987eb1284ad6","Type":"ContainerDied","Data":"5fbfebaa37d99c74631dcc3cd48527b9cb7b4ed64a8f05e146e821e94bbb766f"}
Oct 30 03:24:13.717668 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.717356    4278 generic.go:356] "Generic (PLEG): container finished" podID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerID="d9388cd3431f23ba24d2037dab3aabb6e370f7a449b01bfb8d3f56e4ecd3bea5" exitCode=0
Oct 30 03:24:13.717668 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.717467    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" event={"ID":"4748216d-6793-4ea1-8bc8-e6982cb82707","Type":"ContainerDied","Data":"d9388cd3431f23ba24d2037dab3aabb6e370f7a449b01bfb8d3f56e4ecd3bea5"}
Oct 30 03:24:13.806738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.806704    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:24:13.829541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.829515    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:24:13.854127 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.854104    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:24:13.961488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.961383    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-certs\") pod \"4748216d-6793-4ea1-8bc8-e6982cb82707\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") "
Oct 30 03:24:13.961488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.961450    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-istio-token\" (UniqueName: \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-istio-token\") pod \"9e31519d-e465-4414-b22b-554f662ae87f\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") "
Oct 30 03:24:13.961488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.961480    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-token\" (UniqueName: \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-token\") pod \"4748216d-6793-4ea1-8bc8-e6982cb82707\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") "
Oct 30 03:24:13.961993 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.961526    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-kube-api-access-mtkn7\" (UniqueName: \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-kube-api-access-mtkn7\") pod \"9e31519d-e465-4414-b22b-554f662ae87f\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") "
Oct 30 03:24:13.961993 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.961584    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e31519d-e465-4414-b22b-554f662ae87f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e31519d-e465-4414-b22b-554f662ae87f-istiod-ca-cert\") pod \"9e31519d-e465-4414-b22b-554f662ae87f\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") "
Oct 30 03:24:13.961993 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.961698    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-certs" pod "4748216d-6793-4ea1-8bc8-e6982cb82707" (UID: "4748216d-6793-4ea1-8bc8-e6982cb82707"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.962260 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.962044    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e31519d-e465-4414-b22b-554f662ae87f-istiod-ca-cert" pod "9e31519d-e465-4414-b22b-554f662ae87f" (UID: "9e31519d-e465-4414-b22b-554f662ae87f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:13.963051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.963021    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-socket\") pod \"4748216d-6793-4ea1-8bc8-e6982cb82707\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") "
Oct 30 03:24:13.963746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.963118    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-credential-socket\") pod \"4748216d-6793-4ea1-8bc8-e6982cb82707\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") "
Oct 30 03:24:13.964755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.963930    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-credential-socket\") pod \"9e31519d-e465-4414-b22b-554f662ae87f\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") "
Oct 30 03:24:13.964755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.964480    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-socket" pod "4748216d-6793-4ea1-8bc8-e6982cb82707" (UID: "4748216d-6793-4ea1-8bc8-e6982cb82707"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.964755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.964612    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-credential-socket" pod "9e31519d-e465-4414-b22b-554f662ae87f" (UID: "9e31519d-e465-4414-b22b-554f662ae87f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.964755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.964679    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-token" pod "4748216d-6793-4ea1-8bc8-e6982cb82707" (UID: "4748216d-6793-4ea1-8bc8-e6982cb82707"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.965472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.965379    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-credential-socket" pod "4748216d-6793-4ea1-8bc8-e6982cb82707" (UID: "4748216d-6793-4ea1-8bc8-e6982cb82707"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.966621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.966056    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-credential-socket\") pod \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.966762    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-socket\") pod \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.966803    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-data\") pod \"4748216d-6793-4ea1-8bc8-e6982cb82707\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.966848    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-podinfo\") pod \"4748216d-6793-4ea1-8bc8-e6982cb82707\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.966899    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-certs\") pod \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.966941    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-kube-api-access-rmd4w\" (UniqueName: \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-kube-api-access-rmd4w\") pod \"4748216d-6793-4ea1-8bc8-e6982cb82707\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.966986    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-envoy\") pod \"9e31519d-e465-4414-b22b-554f662ae87f\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967017    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istiod-ca-cert\") pod \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967062    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-certs\") pod \"9e31519d-e465-4414-b22b-554f662ae87f\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967091    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-data\") pod \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967135    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-data\") pod \"9e31519d-e465-4414-b22b-554f662ae87f\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967178    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-envoy\") pod \"4748216d-6793-4ea1-8bc8-e6982cb82707\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967223    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4748216d-6793-4ea1-8bc8-e6982cb82707-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4748216d-6793-4ea1-8bc8-e6982cb82707-istiod-ca-cert\") pod \"4748216d-6793-4ea1-8bc8-e6982cb82707\" (UID: \"4748216d-6793-4ea1-8bc8-e6982cb82707\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967254    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-podinfo\") pod \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967293    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-token\" (UniqueName: \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-token\") pod \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967367    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-envoy\") pod \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967401    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-socket\") pod \"9e31519d-e465-4414-b22b-554f662ae87f\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967457    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-kube-api-access-lwl8f\" (UniqueName: \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-kube-api-access-lwl8f\") pod \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\" (UID: \"137707a1-f8b6-4f13-8eb3-987eb1284ad6\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967573    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e31519d-e465-4414-b22b-554f662ae87f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e31519d-e465-4414-b22b-554f662ae87f-istio-podinfo\") pod \"9e31519d-e465-4414-b22b-554f662ae87f\" (UID: \"9e31519d-e465-4414-b22b-554f662ae87f\") "
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.966599    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-credential-socket" pod "137707a1-f8b6-4f13-8eb3-987eb1284ad6" (UID: "137707a1-f8b6-4f13-8eb3-987eb1284ad6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.967690    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-certs" pod "9e31519d-e465-4414-b22b-554f662ae87f" (UID: "9e31519d-e465-4414-b22b-554f662ae87f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.968023    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-certs" pod "137707a1-f8b6-4f13-8eb3-987eb1284ad6" (UID: "137707a1-f8b6-4f13-8eb3-987eb1284ad6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.968032    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-socket" pod "137707a1-f8b6-4f13-8eb3-987eb1284ad6" (UID: "137707a1-f8b6-4f13-8eb3-987eb1284ad6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.968764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.968255    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-data" pod "4748216d-6793-4ea1-8bc8-e6982cb82707" (UID: "4748216d-6793-4ea1-8bc8-e6982cb82707"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.970059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.969809    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-data" pod "137707a1-f8b6-4f13-8eb3-987eb1284ad6" (UID: "137707a1-f8b6-4f13-8eb3-987eb1284ad6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.970059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.969954    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-data" pod "9e31519d-e465-4414-b22b-554f662ae87f" (UID: "9e31519d-e465-4414-b22b-554f662ae87f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.970275    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-istio-token" pod "9e31519d-e465-4414-b22b-554f662ae87f" (UID: "9e31519d-e465-4414-b22b-554f662ae87f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.970658    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4748216d-6793-4ea1-8bc8-e6982cb82707-istiod-ca-cert" pod "4748216d-6793-4ea1-8bc8-e6982cb82707" (UID: "4748216d-6793-4ea1-8bc8-e6982cb82707"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971445    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-socket" pod "9e31519d-e465-4414-b22b-554f662ae87f" (UID: "9e31519d-e465-4414-b22b-554f662ae87f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971611    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971636    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971652    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971667    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4748216d-6793-4ea1-8bc8-e6982cb82707-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971680    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971694    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971707    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971719    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e31519d-e465-4414-b22b-554f662ae87f-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971732    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971746    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971763    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971777    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971792    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971808    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.971908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971822    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.974078 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.971988    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istiod-ca-cert" pod "137707a1-f8b6-4f13-8eb3-987eb1284ad6" (UID: "137707a1-f8b6-4f13-8eb3-987eb1284ad6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:13.976962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.976928    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-podinfo" pod "137707a1-f8b6-4f13-8eb3-987eb1284ad6" (UID: "137707a1-f8b6-4f13-8eb3-987eb1284ad6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:13.977064 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.977021    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-podinfo" pod "4748216d-6793-4ea1-8bc8-e6982cb82707" (UID: "4748216d-6793-4ea1-8bc8-e6982cb82707"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:13.977560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.977533    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-token" pod "137707a1-f8b6-4f13-8eb3-987eb1284ad6" (UID: "137707a1-f8b6-4f13-8eb3-987eb1284ad6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.977696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.977590    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-kube-api-access-mtkn7" pod "9e31519d-e465-4414-b22b-554f662ae87f" (UID: "9e31519d-e465-4414-b22b-554f662ae87f"). InnerVolumeSpecName "kube-api-access-mtkn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.977696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.977612    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-envoy" pod "4748216d-6793-4ea1-8bc8-e6982cb82707" (UID: "4748216d-6793-4ea1-8bc8-e6982cb82707"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.977696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.977660    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-envoy" pod "137707a1-f8b6-4f13-8eb3-987eb1284ad6" (UID: "137707a1-f8b6-4f13-8eb3-987eb1284ad6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.977890 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.977759    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-kube-api-access-rmd4w" pod "4748216d-6793-4ea1-8bc8-e6982cb82707" (UID: "4748216d-6793-4ea1-8bc8-e6982cb82707"). InnerVolumeSpecName "kube-api-access-rmd4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.978683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.978657    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e31519d-e465-4414-b22b-554f662ae87f-istio-podinfo" pod "9e31519d-e465-4414-b22b-554f662ae87f" (UID: "9e31519d-e465-4414-b22b-554f662ae87f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:13.979236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.979213    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-kube-api-access-lwl8f" pod "137707a1-f8b6-4f13-8eb3-987eb1284ad6" (UID: "137707a1-f8b6-4f13-8eb3-987eb1284ad6"). InnerVolumeSpecName "kube-api-access-lwl8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.979619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:13.979594    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-envoy" pod "9e31519d-e465-4414-b22b-554f662ae87f" (UID: "9e31519d-e465-4414-b22b-554f662ae87f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.073537 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073499    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwl8f\" (UniqueName: \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-kube-api-access-lwl8f\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073537 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073536    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e31519d-e465-4414-b22b-554f662ae87f-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073550    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtkn7\" (UniqueName: \"kubernetes.io/projected/9e31519d-e465-4414-b22b-554f662ae87f-kube-api-access-mtkn7\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073564    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073579    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmd4w\" (UniqueName: \"kubernetes.io/projected/4748216d-6793-4ea1-8bc8-e6982cb82707-kube-api-access-rmd4w\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073593    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073607    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073621    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4748216d-6793-4ea1-8bc8-e6982cb82707-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073633    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073645    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073657    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137707a1-f8b6-4f13-8eb3-987eb1284ad6-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.073758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.073669    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e31519d-e465-4414-b22b-554f662ae87f-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.727917 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.727878    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2" event={"ID":"4748216d-6793-4ea1-8bc8-e6982cb82707","Type":"ContainerDied","Data":"8428e76a3e88302626932c050c34dcee5d0ba03c8582f7ed4f94cf51f0bb6d84"}
Oct 30 03:24:14.727917 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.727899    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"
Oct 30 03:24:14.728562 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.727981    4278 scope.go:122] "RemoveContainer" containerID="d9388cd3431f23ba24d2037dab3aabb6e370f7a449b01bfb8d3f56e4ecd3bea5"
Oct 30 03:24:14.729833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.729804    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff" event={"ID":"9e31519d-e465-4414-b22b-554f662ae87f","Type":"ContainerDied","Data":"6b868d2809c5e6c4606d5a25d9493805c6b35967879d1a1304a6476313b950a5"}
Oct 30 03:24:14.729987 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.729812    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"
Oct 30 03:24:14.733003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.732980    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx" event={"ID":"137707a1-f8b6-4f13-8eb3-987eb1284ad6","Type":"ContainerDied","Data":"c102e42d09f614e658a943d9e4ce53948cfb5f82c5e29ac663c58e4e49bdfb83"}
Oct 30 03:24:14.733115 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.733041    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"
Oct 30 03:24:14.740347 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.740326    4278 scope.go:122] "RemoveContainer" containerID="77b5b5a1aacf1ff2167ed1a2d3c3d2965c9e23debc9d91ce56d2274304e3766d"
Oct 30 03:24:14.750588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.750535    4278 scope.go:122] "RemoveContainer" containerID="5fbfebaa37d99c74631dcc3cd48527b9cb7b4ed64a8f05e146e821e94bbb766f"
Oct 30 03:24:14.799250 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.799214    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"]
Oct 30 03:24:14.809961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.809930    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-gwffx"]
Oct 30 03:24:14.845082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.845051    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"]
Oct 30 03:24:14.853580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.853549    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-qb9ff"]
Oct 30 03:24:14.889006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.888980    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"]
Oct 30 03:24:14.894462 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:14.894407    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-nnbb2"]
Oct 30 03:24:16.203798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:16.203761    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" path="/var/lib/kubelet/pods/137707a1-f8b6-4f13-8eb3-987eb1284ad6/volumes"
Oct 30 03:24:16.204469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:16.204448    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" path="/var/lib/kubelet/pods/4748216d-6793-4ea1-8bc8-e6982cb82707/volumes"
Oct 30 03:24:16.204925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:16.204910    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e31519d-e465-4414-b22b-554f662ae87f" path="/var/lib/kubelet/pods/9e31519d-e465-4414-b22b-554f662ae87f/volumes"
Oct 30 03:24:18.930614 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:18.930578    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"]
Oct 30 03:24:22.762621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:22.762565    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main" containerID="cri-o://0241739926de82216038889c851737df174af388f5e414c928061276b16ce157" gracePeriod=45
Oct 30 03:24:23.165123 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.165093    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 03:24:23.222009 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.221963    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tls-certs\" (UniqueName: \"kubernetes.io/secret/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tls-certs\") pod \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") "
Oct 30 03:24:23.222182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.222041    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-kube-api-access-zpp2z\" (UniqueName: \"kubernetes.io/projected/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-kube-api-access-zpp2z\") pod \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") "
Oct 30 03:24:23.222182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.222076    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tmp-dir\") pod \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") "
Oct 30 03:24:23.222182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.222170    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-home\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-home\") pod \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") "
Oct 30 03:24:23.222380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.222283    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-model-cache\") pod \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") "
Oct 30 03:24:23.222380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.222327    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-dshm\") pod \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\" (UID: \"1212a9d1-03d0-4cf6-ac31-f6768e9e3127\") "
Oct 30 03:24:23.222977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.222929    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-home" pod "1212a9d1-03d0-4cf6-ac31-f6768e9e3127" (UID: "1212a9d1-03d0-4cf6-ac31-f6768e9e3127"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.223138 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.222975    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tmp-dir" pod "1212a9d1-03d0-4cf6-ac31-f6768e9e3127" (UID: "1212a9d1-03d0-4cf6-ac31-f6768e9e3127"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.223311 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.223280    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-model-cache" pod "1212a9d1-03d0-4cf6-ac31-f6768e9e3127" (UID: "1212a9d1-03d0-4cf6-ac31-f6768e9e3127"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.223797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.223773    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.223915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.223902    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.224003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.223992    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.225254 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.225155    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-kube-api-access-zpp2z" pod "1212a9d1-03d0-4cf6-ac31-f6768e9e3127" (UID: "1212a9d1-03d0-4cf6-ac31-f6768e9e3127"). InnerVolumeSpecName "kube-api-access-zpp2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:23.225574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.225548    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-dshm" pod "1212a9d1-03d0-4cf6-ac31-f6768e9e3127" (UID: "1212a9d1-03d0-4cf6-ac31-f6768e9e3127"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.225831 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.225622    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tls-certs" pod "1212a9d1-03d0-4cf6-ac31-f6768e9e3127" (UID: "1212a9d1-03d0-4cf6-ac31-f6768e9e3127"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:23.325724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.325695    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.325724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.325723    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpp2z\" (UniqueName: \"kubernetes.io/projected/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-kube-api-access-zpp2z\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.325940 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.325733    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1212a9d1-03d0-4cf6-ac31-f6768e9e3127-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.731257 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.731221    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"]
Oct 30 03:24:23.732176 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732150    4278 cpu_manager.go:414] "containerMap: removing container" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerName="istio-proxy"
Oct 30 03:24:23.732176 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732176    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerName="istio-proxy"
Oct 30 03:24:23.732370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732196    4278 cpu_manager.go:414] "containerMap: removing container" podUID="cd1579bf-0969-4f13-9850-7f56e08c1cc2" containerName="revoke-keys"
Oct 30 03:24:23.732370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732206    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1579bf-0969-4f13-9850-7f56e08c1cc2" containerName="revoke-keys"
Oct 30 03:24:23.732370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732233    4278 cpu_manager.go:414] "containerMap: removing container" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main"
Oct 30 03:24:23.732370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732241    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main"
Oct 30 03:24:23.732370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732252    4278 cpu_manager.go:414] "containerMap: removing container" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy"
Oct 30 03:24:23.732370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732259    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy"
Oct 30 03:24:23.732370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732313    4278 cpu_manager.go:414] "containerMap: removing container" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy"
Oct 30 03:24:23.732370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732320    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy"
Oct 30 03:24:23.732893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732520    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="4748216d-6793-4ea1-8bc8-e6982cb82707" containerName="istio-proxy"
Oct 30 03:24:23.732893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732540    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="137707a1-f8b6-4f13-8eb3-987eb1284ad6" containerName="istio-proxy"
Oct 30 03:24:23.732893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732555    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1579bf-0969-4f13-9850-7f56e08c1cc2" containerName="revoke-keys"
Oct 30 03:24:23.732893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732567    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerName="main"
Oct 30 03:24:23.732893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.732577    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e31519d-e465-4414-b22b-554f662ae87f" containerName="istio-proxy"
Oct 30 03:24:23.736810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.736787    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:23.762519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.762488    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"]
Oct 30 03:24:23.769192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.769164    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:24:23.769628 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.769269    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:24:23.822392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.822358    4278 generic.go:356] "Generic (PLEG): container finished" podID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" containerID="0241739926de82216038889c851737df174af388f5e414c928061276b16ce157" exitCode=0
Oct 30 03:24:23.823011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.822688    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"
Oct 30 03:24:23.823011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.822681    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" event={"ID":"1212a9d1-03d0-4cf6-ac31-f6768e9e3127","Type":"ContainerDied","Data":"0241739926de82216038889c851737df174af388f5e414c928061276b16ce157"}
Oct 30 03:24:23.823011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.822948    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d" event={"ID":"1212a9d1-03d0-4cf6-ac31-f6768e9e3127","Type":"ContainerDied","Data":"a6233f23cda5f044193a5198a14d86c72643b002af88f09712a71b58ba0b3fd5"}
Oct 30 03:24:23.823189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.823025    4278 scope.go:122] "RemoveContainer" containerID="0241739926de82216038889c851737df174af388f5e414c928061276b16ce157"
Oct 30 03:24:23.834969 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.834943    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mps7b\" (UniqueName: \"kubernetes.io/projected/03720fc6-d91b-40f9-9f4e-db5a047a04ec-kube-api-access-mps7b\") pod \"maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:23.835071 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.834986    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:23.835165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.835148    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:23.839797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.839770    4278 scope.go:122] "RemoveContainer" containerID="0241739926de82216038889c851737df174af388f5e414c928061276b16ce157"
Oct 30 03:24:23.840264 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:24:23.840231    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0241739926de82216038889c851737df174af388f5e414c928061276b16ce157\": container with ID starting with 0241739926de82216038889c851737df174af388f5e414c928061276b16ce157 not found: ID does not exist" containerID="0241739926de82216038889c851737df174af388f5e414c928061276b16ce157"
Oct 30 03:24:23.840359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.840276    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0241739926de82216038889c851737df174af388f5e414c928061276b16ce157"} err="failed to get container status \"0241739926de82216038889c851737df174af388f5e414c928061276b16ce157\": rpc error: code = NotFound desc = could not find container \"0241739926de82216038889c851737df174af388f5e414c928061276b16ce157\": container with ID starting with 0241739926de82216038889c851737df174af388f5e414c928061276b16ce157 not found: ID does not exist"
Oct 30 03:24:23.887324 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.887297    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"]
Oct 30 03:24:23.891374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.891345    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-810119/premium-w2-810119-kserve-6cff7b7ccd-44m5d"]
Oct 30 03:24:23.936569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.936530    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:23.936792 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.936771    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mps7b\" (UniqueName: \"kubernetes.io/projected/03720fc6-d91b-40f9-9f4e-db5a047a04ec-kube-api-access-mps7b\") pod \"maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:23.936870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.936802    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:23.937501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.937479    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:23.939252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.939216    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:23.960785 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:23.960666    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mps7b\" (UniqueName: \"kubernetes.io/projected/03720fc6-d91b-40f9-9f4e-db5a047a04ec-kube-api-access-mps7b\") pod \"maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:24.053577 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:24.053539    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:24.196456 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:24:24.196405    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:24:24.201291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:24.201260    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1212a9d1-03d0-4cf6-ac31-f6768e9e3127" path="/var/lib/kubelet/pods/1212a9d1-03d0-4cf6-ac31-f6768e9e3127/volumes"
Oct 30 03:24:24.376941 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:24.376914    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"]
Oct 30 03:24:24.378150 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:24:24.378121    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03720fc6_d91b_40f9_9f4e_db5a047a04ec.slice/crio-9c51b82b26b53085087c37c144bbe38e7bc3f7bb01f61d96c9fb9aba87b73a02 WatchSource:0}: Error finding container 9c51b82b26b53085087c37c144bbe38e7bc3f7bb01f61d96c9fb9aba87b73a02: Status 404 returned error can't find the container with id 9c51b82b26b53085087c37c144bbe38e7bc3f7bb01f61d96c9fb9aba87b73a02
Oct 30 03:24:24.834076 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:24.834038    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5" event={"ID":"03720fc6-d91b-40f9-9f4e-db5a047a04ec","Type":"ContainerStarted","Data":"79ad6bf0348b25c6dbf54033cab6132b6619ed8de1929c94e965b5ccb5fff7f2"}
Oct 30 03:24:24.834573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:24.834094    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5" event={"ID":"03720fc6-d91b-40f9-9f4e-db5a047a04ec","Type":"ContainerStarted","Data":"9c51b82b26b53085087c37c144bbe38e7bc3f7bb01f61d96c9fb9aba87b73a02"}
Oct 30 03:24:24.883125 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:24.883074    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5" podStartSLOduration=1.883060052 podStartE2EDuration="1.883060052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:24.880939548 +0000 UTC m=+2602.805147737" watchObservedRunningTime="2026-10-30 03:24:24.883060052 +0000 UTC m=+2602.807268241"
Oct 30 03:24:25.843236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:25.843204    4278 generic.go:356] "Generic (PLEG): container finished" podID="03720fc6-d91b-40f9-9f4e-db5a047a04ec" containerID="79ad6bf0348b25c6dbf54033cab6132b6619ed8de1929c94e965b5ccb5fff7f2" exitCode=0
Oct 30 03:24:25.843731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:25.843297    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5" event={"ID":"03720fc6-d91b-40f9-9f4e-db5a047a04ec","Type":"ContainerDied","Data":"79ad6bf0348b25c6dbf54033cab6132b6619ed8de1929c94e965b5ccb5fff7f2"}
Oct 30 03:24:27.009601 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.009572    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:27.079025 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.078987    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-openshift-ca\") pod \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") "
Oct 30 03:24:27.079201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.079051    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-xks-ca\") pod \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") "
Oct 30 03:24:27.079201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.079137    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03720fc6-d91b-40f9-9f4e-db5a047a04ec-kube-api-access-mps7b\" (UniqueName: \"kubernetes.io/projected/03720fc6-d91b-40f9-9f4e-db5a047a04ec-kube-api-access-mps7b\") pod \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\" (UID: \"03720fc6-d91b-40f9-9f4e-db5a047a04ec\") "
Oct 30 03:24:27.079459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.079389    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-openshift-ca" pod "03720fc6-d91b-40f9-9f4e-db5a047a04ec" (UID: "03720fc6-d91b-40f9-9f4e-db5a047a04ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:27.079857 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.079833    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:27.081363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.081338    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03720fc6-d91b-40f9-9f4e-db5a047a04ec-kube-api-access-mps7b" pod "03720fc6-d91b-40f9-9f4e-db5a047a04ec" (UID: "03720fc6-d91b-40f9-9f4e-db5a047a04ec"). InnerVolumeSpecName "kube-api-access-mps7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:27.081737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.081713    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-xks-ca" pod "03720fc6-d91b-40f9-9f4e-db5a047a04ec" (UID: "03720fc6-d91b-40f9-9f4e-db5a047a04ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:27.181590 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.181507    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mps7b\" (UniqueName: \"kubernetes.io/projected/03720fc6-d91b-40f9-9f4e-db5a047a04ec-kube-api-access-mps7b\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:27.181590 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.181540    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03720fc6-d91b-40f9-9f4e-db5a047a04ec-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:27.859587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.859554    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5" event={"ID":"03720fc6-d91b-40f9-9f4e-db5a047a04ec","Type":"ContainerDied","Data":"9c51b82b26b53085087c37c144bbe38e7bc3f7bb01f61d96c9fb9aba87b73a02"}
Oct 30 03:24:27.859791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.859603    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c51b82b26b53085087c37c144bbe38e7bc3f7bb01f61d96c9fb9aba87b73a02"
Oct 30 03:24:27.859791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:27.859607    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"
Oct 30 03:24:29.054680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:29.054642    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"]
Oct 30 03:24:29.067104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:29.067073    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682dd073-dtc6f"]
Oct 30 03:24:29.945853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:29.945819    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"]
Oct 30 03:24:30.226438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:30.226343    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba7d411-d429-4dd7-b9f9-517d228c399f" path="/var/lib/kubelet/pods/4ba7d411-d429-4dd7-b9f9-517d228c399f/volumes"
Oct 30 03:24:33.966847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:33.966806    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8" podUID="9ed1592a-f1eb-457d-a101-50017d30b4d5" containerName="main" containerID="cri-o://4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59" gracePeriod=45
Oct 30 03:24:34.275623 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:34.275581    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8" podUID="9ed1592a-f1eb-457d-a101-50017d30b4d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": dial tcp 10.134.0.212:8000: connect: connection refused"
Oct 30 03:24:34.565836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:34.565798    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"]
Oct 30 03:24:34.577640 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:34.576699    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-360818-w57jr"]
Oct 30 03:24:34.948393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:34.947136    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd" podUID="b94b1958-4b28-4e6d-8249-3dfe0b8507f2" containerName="payload-pre-processing" containerID="cri-o://ea3bec75cae712f37b20d3f501721b5c025c149ea587bf5d8224252caa3ecd14" gracePeriod=25
Oct 30 03:24:35.126933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.126870    4278 generic.go:356] "Generic (PLEG): container finished" podID="b94b1958-4b28-4e6d-8249-3dfe0b8507f2" containerID="ea3bec75cae712f37b20d3f501721b5c025c149ea587bf5d8224252caa3ecd14" exitCode=0
Oct 30 03:24:35.130766 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.130679    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd" event={"ID":"b94b1958-4b28-4e6d-8249-3dfe0b8507f2","Type":"ContainerDied","Data":"ea3bec75cae712f37b20d3f501721b5c025c149ea587bf5d8224252caa3ecd14"}
Oct 30 03:24:35.133029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.132989    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:24:35.134284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.134220    4278 generic.go:356] "Generic (PLEG): container finished" podID="9ed1592a-f1eb-457d-a101-50017d30b4d5" containerID="4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59" exitCode=0
Oct 30 03:24:35.134567 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.134531    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8" event={"ID":"9ed1592a-f1eb-457d-a101-50017d30b4d5","Type":"ContainerDied","Data":"4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59"}
Oct 30 03:24:35.134733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.134674    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8" event={"ID":"9ed1592a-f1eb-457d-a101-50017d30b4d5","Type":"ContainerDied","Data":"000ec973026842950cc01387e3ec1fad3556276d821c33f5f96f3615a22a85fb"}
Oct 30 03:24:35.134733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.134699    4278 scope.go:122] "RemoveContainer" containerID="4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59"
Oct 30 03:24:35.154530 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.154509    4278 scope.go:122] "RemoveContainer" containerID="4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59"
Oct 30 03:24:35.155083 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:24:35.155056    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59\": container with ID starting with 4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59 not found: ID does not exist" containerID="4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59"
Oct 30 03:24:35.155198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.155089    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59"} err="failed to get container status \"4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59\": rpc error: code = NotFound desc = could not find container \"4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59\": container with ID starting with 4c41a2d728ae83dcc1bd1f6223613d129cfe7144d257de81295f89881d210a59 not found: ID does not exist"
Oct 30 03:24:35.211449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.210916    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed1592a-f1eb-457d-a101-50017d30b4d5-kube-api-access-x9z7g\" (UniqueName: \"kubernetes.io/projected/9ed1592a-f1eb-457d-a101-50017d30b4d5-kube-api-access-x9z7g\") pod \"9ed1592a-f1eb-457d-a101-50017d30b4d5\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") "
Oct 30 03:24:35.211449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.211006    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-tmp-dir\") pod \"9ed1592a-f1eb-457d-a101-50017d30b4d5\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") "
Oct 30 03:24:35.211449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.211120    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-home\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-home\") pod \"9ed1592a-f1eb-457d-a101-50017d30b4d5\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") "
Oct 30 03:24:35.211449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.211152    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed1592a-f1eb-457d-a101-50017d30b4d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed1592a-f1eb-457d-a101-50017d30b4d5-tls-certs\") pod \"9ed1592a-f1eb-457d-a101-50017d30b4d5\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") "
Oct 30 03:24:35.211449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.211191    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-model-cache\") pod \"9ed1592a-f1eb-457d-a101-50017d30b4d5\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") "
Oct 30 03:24:35.211449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.211235    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-dshm\") pod \"9ed1592a-f1eb-457d-a101-50017d30b4d5\" (UID: \"9ed1592a-f1eb-457d-a101-50017d30b4d5\") "
Oct 30 03:24:35.217443 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.214692    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-home" pod "9ed1592a-f1eb-457d-a101-50017d30b4d5" (UID: "9ed1592a-f1eb-457d-a101-50017d30b4d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:35.217443 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.215209    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-model-cache" pod "9ed1592a-f1eb-457d-a101-50017d30b4d5" (UID: "9ed1592a-f1eb-457d-a101-50017d30b4d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:35.217443 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.215394    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-tmp-dir" pod "9ed1592a-f1eb-457d-a101-50017d30b4d5" (UID: "9ed1592a-f1eb-457d-a101-50017d30b4d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:35.238449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.236662    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed1592a-f1eb-457d-a101-50017d30b4d5-kube-api-access-x9z7g" pod "9ed1592a-f1eb-457d-a101-50017d30b4d5" (UID: "9ed1592a-f1eb-457d-a101-50017d30b4d5"). InnerVolumeSpecName "kube-api-access-x9z7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:35.238449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.236816    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-dshm" pod "9ed1592a-f1eb-457d-a101-50017d30b4d5" (UID: "9ed1592a-f1eb-457d-a101-50017d30b4d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:35.239618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.239274    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed1592a-f1eb-457d-a101-50017d30b4d5-tls-certs" pod "9ed1592a-f1eb-457d-a101-50017d30b4d5" (UID: "9ed1592a-f1eb-457d-a101-50017d30b4d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:35.325860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.325741    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9z7g\" (UniqueName: \"kubernetes.io/projected/9ed1592a-f1eb-457d-a101-50017d30b4d5-kube-api-access-x9z7g\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:35.325860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.325775    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:35.325860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.325792    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:35.325860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.325805    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed1592a-f1eb-457d-a101-50017d30b4d5-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:35.325860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.325820    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:35.325860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.325832    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed1592a-f1eb-457d-a101-50017d30b4d5-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:35.644475 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.643069    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:24:35.738499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.738192    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-kube-api-access-46bvl\" (UniqueName: \"kubernetes.io/projected/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-kube-api-access-46bvl\") pod \"b94b1958-4b28-4e6d-8249-3dfe0b8507f2\" (UID: \"b94b1958-4b28-4e6d-8249-3dfe0b8507f2\") "
Oct 30 03:24:35.738499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.738265    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-plugins-config-volume\") pod \"b94b1958-4b28-4e6d-8249-3dfe0b8507f2\" (UID: \"b94b1958-4b28-4e6d-8249-3dfe0b8507f2\") "
Oct 30 03:24:35.739756 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.739321    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-plugins-config-volume" pod "b94b1958-4b28-4e6d-8249-3dfe0b8507f2" (UID: "b94b1958-4b28-4e6d-8249-3dfe0b8507f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:35.753752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.753377    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-kube-api-access-46bvl" pod "b94b1958-4b28-4e6d-8249-3dfe0b8507f2" (UID: "b94b1958-4b28-4e6d-8249-3dfe0b8507f2"). InnerVolumeSpecName "kube-api-access-46bvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:35.844892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.844823    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46bvl\" (UniqueName: \"kubernetes.io/projected/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-kube-api-access-46bvl\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:35.844892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:35.844862    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94b1958-4b28-4e6d-8249-3dfe0b8507f2-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:36.143979 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.143538    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd" event={"ID":"b94b1958-4b28-4e6d-8249-3dfe0b8507f2","Type":"ContainerDied","Data":"098085428f3b97ef4e03a77d30f171b20007ebac873da2324ad9a0531eb1b14c"}
Oct 30 03:24:36.144523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.144081    4278 scope.go:122] "RemoveContainer" containerID="ea3bec75cae712f37b20d3f501721b5c025c149ea587bf5d8224252caa3ecd14"
Oct 30 03:24:36.144523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.144111    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"
Oct 30 03:24:36.144992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.144901    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"
Oct 30 03:24:36.234822 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.234748    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1579bf-0969-4f13-9850-7f56e08c1cc2" path="/var/lib/kubelet/pods/cd1579bf-0969-4f13-9850-7f56e08c1cc2/volumes"
Oct 30 03:24:36.235107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.235090    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"]
Oct 30 03:24:36.235833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.235803    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-6b5f19/auth-test-model-6b5f19-kserve-7675cc6649-88sb8"]
Oct 30 03:24:36.306451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.305479    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"]
Oct 30 03:24:36.325115 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.325084    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-584cd8fdbc-x49rd"]
Oct 30 03:24:36.624759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.624720    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"]
Oct 30 03:24:36.625606 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.625585    4278 cpu_manager.go:414] "containerMap: removing container" podUID="03720fc6-d91b-40f9-9f4e-db5a047a04ec" containerName="revoke-keys"
Oct 30 03:24:36.625606 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.625605    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03720fc6-d91b-40f9-9f4e-db5a047a04ec" containerName="revoke-keys"
Oct 30 03:24:36.625802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.625638    4278 cpu_manager.go:414] "containerMap: removing container" podUID="9ed1592a-f1eb-457d-a101-50017d30b4d5" containerName="main"
Oct 30 03:24:36.625802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.625646    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed1592a-f1eb-457d-a101-50017d30b4d5" containerName="main"
Oct 30 03:24:36.625802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.625665    4278 cpu_manager.go:414] "containerMap: removing container" podUID="b94b1958-4b28-4e6d-8249-3dfe0b8507f2" containerName="payload-pre-processing"
Oct 30 03:24:36.625802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.625674    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94b1958-4b28-4e6d-8249-3dfe0b8507f2" containerName="payload-pre-processing"
Oct 30 03:24:36.626022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.625826    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed1592a-f1eb-457d-a101-50017d30b4d5" containerName="main"
Oct 30 03:24:36.626022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.625837    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94b1958-4b28-4e6d-8249-3dfe0b8507f2" containerName="payload-pre-processing"
Oct 30 03:24:36.626022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.625857    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="03720fc6-d91b-40f9-9f4e-db5a047a04ec" containerName="revoke-keys"
Oct 30 03:24:36.635958 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.635931    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:36.639406 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.639384    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"]
Oct 30 03:24:36.645307 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.643693    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"]
Oct 30 03:24:36.648697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.648675    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:36.653659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.652992    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:24:36.655277 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.655259    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:24:36.722225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.722139    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"]
Oct 30 03:24:36.756819 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.756775    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02df6932-hbdh7\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:36.757029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.757000    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ee4798-zr2xl\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:36.757124 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.757107    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02df6932-hbdh7\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:36.757193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.757177    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ee4798-zr2xl\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:36.757256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.757231    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwnqg\" (UniqueName: \"kubernetes.io/projected/838a3dda-fd7f-404b-af80-0cf69627af89-kube-api-access-kwnqg\") pod \"maas-api-revoke-sub-f2ee4798-zr2xl\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:36.757312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.757303    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml6mz\" (UniqueName: \"kubernetes.io/projected/c4a62072-3a62-4e1a-b33b-7a198b372c04-kube-api-access-ml6mz\") pod \"maas-api-revoke-sub-02df6932-hbdh7\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:36.910494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.872598    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ee4798-zr2xl\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:36.910494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.872734    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02df6932-hbdh7\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:36.910494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.872810    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ee4798-zr2xl\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:36.910494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.872868    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwnqg\" (UniqueName: \"kubernetes.io/projected/838a3dda-fd7f-404b-af80-0cf69627af89-kube-api-access-kwnqg\") pod \"maas-api-revoke-sub-f2ee4798-zr2xl\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:36.910494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.872945    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml6mz\" (UniqueName: \"kubernetes.io/projected/c4a62072-3a62-4e1a-b33b-7a198b372c04-kube-api-access-ml6mz\") pod \"maas-api-revoke-sub-02df6932-hbdh7\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:36.910494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.872991    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02df6932-hbdh7\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:36.910494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.873958    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02df6932-hbdh7\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:36.910494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:36.875022    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ee4798-zr2xl\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:37.104457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:37.084561    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwnqg\" (UniqueName: \"kubernetes.io/projected/838a3dda-fd7f-404b-af80-0cf69627af89-kube-api-access-kwnqg\") pod \"maas-api-revoke-sub-f2ee4798-zr2xl\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:37.106798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:37.106732    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02df6932-hbdh7\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:37.115268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:37.115192    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml6mz\" (UniqueName: \"kubernetes.io/projected/c4a62072-3a62-4e1a-b33b-7a198b372c04-kube-api-access-ml6mz\") pod \"maas-api-revoke-sub-02df6932-hbdh7\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:37.136856 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:37.136777    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ee4798-zr2xl\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:37.324493 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:37.324459    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:37.336393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:37.336364    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:37.715856 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:37.715159    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"]
Oct 30 03:24:37.815083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:37.815048    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"]
Oct 30 03:24:37.815705 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:24:37.815353    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a62072_3a62_4e1a_b33b_7a198b372c04.slice/crio-e4b3b221e5d291dd86e6b68c69995a80a833e56d27bdcf064235ef44876dda29 WatchSource:0}: Error finding container e4b3b221e5d291dd86e6b68c69995a80a833e56d27bdcf064235ef44876dda29: Status 404 returned error can't find the container with id e4b3b221e5d291dd86e6b68c69995a80a833e56d27bdcf064235ef44876dda29
Oct 30 03:24:38.201481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:38.201454    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed1592a-f1eb-457d-a101-50017d30b4d5" path="/var/lib/kubelet/pods/9ed1592a-f1eb-457d-a101-50017d30b4d5/volumes"
Oct 30 03:24:38.202100 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:38.202082    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94b1958-4b28-4e6d-8249-3dfe0b8507f2" path="/var/lib/kubelet/pods/b94b1958-4b28-4e6d-8249-3dfe0b8507f2/volumes"
Oct 30 03:24:38.234895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:38.234865    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl" event={"ID":"838a3dda-fd7f-404b-af80-0cf69627af89","Type":"ContainerStarted","Data":"252209b46b2bf9712c66c6f2b521186c168fc11a99cc57a951df28b682ff86cf"}
Oct 30 03:24:38.236752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:38.236724    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7" event={"ID":"c4a62072-3a62-4e1a-b33b-7a198b372c04","Type":"ContainerStarted","Data":"7126ee82c494de2da84eef14b4a3b7b447162aab54e2dcf0b49359a256573631"}
Oct 30 03:24:38.236883 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:38.236778    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7" event={"ID":"c4a62072-3a62-4e1a-b33b-7a198b372c04","Type":"ContainerStarted","Data":"e4b3b221e5d291dd86e6b68c69995a80a833e56d27bdcf064235ef44876dda29"}
Oct 30 03:24:38.291631 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:38.291568    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7" podStartSLOduration=2.291549777 podStartE2EDuration="2.291549777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:38.289130081 +0000 UTC m=+2616.213338271" watchObservedRunningTime="2026-10-30 03:24:38.291549777 +0000 UTC m=+2616.215757966"
Oct 30 03:24:39.042597 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:39.042564    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"]
Oct 30 03:24:39.055272 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:39.055242    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd0a039-bb5mn"]
Oct 30 03:24:39.250485 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:39.250387    4278 generic.go:356] "Generic (PLEG): container finished" podID="c4a62072-3a62-4e1a-b33b-7a198b372c04" containerID="7126ee82c494de2da84eef14b4a3b7b447162aab54e2dcf0b49359a256573631" exitCode=0
Oct 30 03:24:39.250645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:39.250476    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7" event={"ID":"c4a62072-3a62-4e1a-b33b-7a198b372c04","Type":"ContainerDied","Data":"7126ee82c494de2da84eef14b4a3b7b447162aab54e2dcf0b49359a256573631"}
Oct 30 03:24:39.252712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:39.252680    4278 generic.go:356] "Generic (PLEG): container finished" podID="838a3dda-fd7f-404b-af80-0cf69627af89" containerID="b297ec6c2fb4c5df7a1343d0babff859b40d3f23cf8cdfc58fa070f4641b7ae1" exitCode=0
Oct 30 03:24:39.252712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:39.252726    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl" event={"ID":"838a3dda-fd7f-404b-af80-0cf69627af89","Type":"ContainerDied","Data":"b297ec6c2fb4c5df7a1343d0babff859b40d3f23cf8cdfc58fa070f4641b7ae1"}
Oct 30 03:24:40.200557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.200523    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95cae793-895e-4bb0-b884-443c42d4b0f2" path="/var/lib/kubelet/pods/95cae793-895e-4bb0-b884-443c42d4b0f2/volumes"
Oct 30 03:24:40.476909 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.476880    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:40.505371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.505345    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:40.578262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.576997    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-openshift-ca\") pod \"838a3dda-fd7f-404b-af80-0cf69627af89\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") "
Oct 30 03:24:40.578262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.577104    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a62072-3a62-4e1a-b33b-7a198b372c04-kube-api-access-ml6mz\" (UniqueName: \"kubernetes.io/projected/c4a62072-3a62-4e1a-b33b-7a198b372c04-kube-api-access-ml6mz\") pod \"c4a62072-3a62-4e1a-b33b-7a198b372c04\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") "
Oct 30 03:24:40.578262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.577135    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838a3dda-fd7f-404b-af80-0cf69627af89-kube-api-access-kwnqg\" (UniqueName: \"kubernetes.io/projected/838a3dda-fd7f-404b-af80-0cf69627af89-kube-api-access-kwnqg\") pod \"838a3dda-fd7f-404b-af80-0cf69627af89\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") "
Oct 30 03:24:40.578262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.577362    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-xks-ca\") pod \"c4a62072-3a62-4e1a-b33b-7a198b372c04\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") "
Oct 30 03:24:40.578262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.577440    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-openshift-ca\") pod \"c4a62072-3a62-4e1a-b33b-7a198b372c04\" (UID: \"c4a62072-3a62-4e1a-b33b-7a198b372c04\") "
Oct 30 03:24:40.578262 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.577503    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-xks-ca\") pod \"838a3dda-fd7f-404b-af80-0cf69627af89\" (UID: \"838a3dda-fd7f-404b-af80-0cf69627af89\") "
Oct 30 03:24:40.583473 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.579804    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-openshift-ca" pod "838a3dda-fd7f-404b-af80-0cf69627af89" (UID: "838a3dda-fd7f-404b-af80-0cf69627af89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:40.583473 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.580052    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-openshift-ca" pod "c4a62072-3a62-4e1a-b33b-7a198b372c04" (UID: "c4a62072-3a62-4e1a-b33b-7a198b372c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:40.597515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.587280    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a62072-3a62-4e1a-b33b-7a198b372c04-kube-api-access-ml6mz" pod "c4a62072-3a62-4e1a-b33b-7a198b372c04" (UID: "c4a62072-3a62-4e1a-b33b-7a198b372c04"). InnerVolumeSpecName "kube-api-access-ml6mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:40.597515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.590808    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-xks-ca" pod "c4a62072-3a62-4e1a-b33b-7a198b372c04" (UID: "c4a62072-3a62-4e1a-b33b-7a198b372c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:40.601213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.601180    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-xks-ca" pod "838a3dda-fd7f-404b-af80-0cf69627af89" (UID: "838a3dda-fd7f-404b-af80-0cf69627af89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:40.603245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.603220    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838a3dda-fd7f-404b-af80-0cf69627af89-kube-api-access-kwnqg" pod "838a3dda-fd7f-404b-af80-0cf69627af89" (UID: "838a3dda-fd7f-404b-af80-0cf69627af89"). InnerVolumeSpecName "kube-api-access-kwnqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:40.680226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.680188    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:40.680226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.680229    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml6mz\" (UniqueName: \"kubernetes.io/projected/c4a62072-3a62-4e1a-b33b-7a198b372c04-kube-api-access-ml6mz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:40.680419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.680247    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwnqg\" (UniqueName: \"kubernetes.io/projected/838a3dda-fd7f-404b-af80-0cf69627af89-kube-api-access-kwnqg\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:40.680419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.680260    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:40.680419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.680274    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a62072-3a62-4e1a-b33b-7a198b372c04-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:40.680419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:40.680288    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/838a3dda-fd7f-404b-af80-0cf69627af89-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:41.273910 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:41.273883    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"
Oct 30 03:24:41.274413 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:41.273904    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl" event={"ID":"838a3dda-fd7f-404b-af80-0cf69627af89","Type":"ContainerDied","Data":"252209b46b2bf9712c66c6f2b521186c168fc11a99cc57a951df28b682ff86cf"}
Oct 30 03:24:41.274413 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:41.273968    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="252209b46b2bf9712c66c6f2b521186c168fc11a99cc57a951df28b682ff86cf"
Oct 30 03:24:41.275716 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:41.275693    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7" event={"ID":"c4a62072-3a62-4e1a-b33b-7a198b372c04","Type":"ContainerDied","Data":"e4b3b221e5d291dd86e6b68c69995a80a833e56d27bdcf064235ef44876dda29"}
Oct 30 03:24:41.275846 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:41.275739    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4b3b221e5d291dd86e6b68c69995a80a833e56d27bdcf064235ef44876dda29"
Oct 30 03:24:41.275846 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:41.275799    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"
Oct 30 03:24:42.781449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:42.762524    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"]
Oct 30 03:24:42.781449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:42.763026    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" containerName="istio-proxy" containerID="cri-o://16cf572a8490a2d705d1598ab9fe2529a85b2b304d8c37997f738b80e3c65daf" gracePeriod=30
Oct 30 03:24:42.781449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:42.775167    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"]
Oct 30 03:24:42.781449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:42.775449    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy" containerID="cri-o://bfe4bc1548fa5afe0a187b42ebae4f8a6da7824a3cf995bf9b2447807354c0d9" gracePeriod=30
Oct 30 03:24:42.809917 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:42.809878    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:24:42.810183 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:42.810144    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:24:43.293781 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.293741    4278 generic.go:356] "Generic (PLEG): container finished" podID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerID="83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4" exitCode=0
Oct 30 03:24:43.293999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.293777    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" event={"ID":"db05207e-7e87-4757-a68c-e9bf54b970fc","Type":"ContainerDied","Data":"83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4"}
Oct 30 03:24:43.294265 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.294247    4278 scope.go:122] "RemoveContainer" containerID="83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4"
Oct 30 03:24:43.295955 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.295926    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:43.296055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.295998    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:43.296055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.296029    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:43.394946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.394913    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"]
Oct 30 03:24:43.395496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.395479    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c4a62072-3a62-4e1a-b33b-7a198b372c04" containerName="revoke-keys"
Oct 30 03:24:43.395566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.395504    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a62072-3a62-4e1a-b33b-7a198b372c04" containerName="revoke-keys"
Oct 30 03:24:43.395566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.395538    4278 cpu_manager.go:414] "containerMap: removing container" podUID="838a3dda-fd7f-404b-af80-0cf69627af89" containerName="revoke-keys"
Oct 30 03:24:43.395566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.395545    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838a3dda-fd7f-404b-af80-0cf69627af89" containerName="revoke-keys"
Oct 30 03:24:43.395687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.395676    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a62072-3a62-4e1a-b33b-7a198b372c04" containerName="revoke-keys"
Oct 30 03:24:43.395724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.395693    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="838a3dda-fd7f-404b-af80-0cf69627af89" containerName="revoke-keys"
Oct 30 03:24:43.431449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.430818    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"]
Oct 30 03:24:43.431449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.430943    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:43.448205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.448185    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:24:43.448455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.448441    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:24:43.522780 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.522728    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:43.522944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.522885    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:43.523080 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.523064    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffg9b\" (UniqueName: \"kubernetes.io/projected/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-kube-api-access-ffg9b\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:43.624306 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.624272    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:43.626518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.624371    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:43.626518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.624561    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffg9b\" (UniqueName: \"kubernetes.io/projected/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-kube-api-access-ffg9b\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:43.626518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.625070    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:43.627117 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.627088    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:43.647890 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.647853    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffg9b\" (UniqueName: \"kubernetes.io/projected/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-kube-api-access-ffg9b\") pod \"maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:43.804483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:43.804451    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:44.120519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:44.120491    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"]
Oct 30 03:24:44.121838 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:24:44.121808    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff426c1_cd0d_465d_8dbf_4a8b2215d99d.slice/crio-49a5f81f537366f99ab5d28e83a93801f810b2db070992733e27b7c802ec240a WatchSource:0}: Error finding container 49a5f81f537366f99ab5d28e83a93801f810b2db070992733e27b7c802ec240a: Status 404 returned error can't find the container with id 49a5f81f537366f99ab5d28e83a93801f810b2db070992733e27b7c802ec240a
Oct 30 03:24:44.330646 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:44.330606    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq" event={"ID":"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d","Type":"ContainerStarted","Data":"49a5f81f537366f99ab5d28e83a93801f810b2db070992733e27b7c802ec240a"}
Oct 30 03:24:44.332636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:44.332607    4278 generic.go:356] "Generic (PLEG): container finished" podID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerID="0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e" exitCode=0
Oct 30 03:24:44.332789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:44.332642    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" event={"ID":"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d","Type":"ContainerDied","Data":"0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e"}
Oct 30 03:24:44.333128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:44.333110    4278 scope.go:122] "RemoveContainer" containerID="0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e"
Oct 30 03:24:44.335175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:44.335140    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:44.335302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:44.335222    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:44.335302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:44.335257    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:44.335489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:44.335308    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" event={"ID":"db05207e-7e87-4757-a68c-e9bf54b970fc","Type":"ContainerStarted","Data":"d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1"}
Oct 30 03:24:45.129461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:45.129067    4278 patch_prober.go:29] interesting pod/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:24:45.129461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:45.129128    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 03:24:45.129461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:45.129407    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:24:45.347643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:45.347606    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq" event={"ID":"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d","Type":"ContainerStarted","Data":"1c618083b57acd80e74b7ac2e7ce8b5ae9dc508ac6137265063e03fde7d340f0"}
Oct 30 03:24:45.350405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:45.350375    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" event={"ID":"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d","Type":"ContainerStarted","Data":"f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab"}
Oct 30 03:24:45.407297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:45.407232    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq" podStartSLOduration=2.407213103 podStartE2EDuration="2.407213103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:45.405795144 +0000 UTC m=+2623.330003337" watchObservedRunningTime="2026-10-30 03:24:45.407213103 +0000 UTC m=+2623.331421293"
Oct 30 03:24:45.812449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:45.811886    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:24:45.829350 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:45.825128    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 03:24:45.829350 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:45.825206    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 03:24:46.150683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:46.148613    4278 patch_prober.go:29] interesting pod/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:24:46.150683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:46.148671    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 03:24:46.359165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:46.359111    4278 generic.go:356] "Generic (PLEG): container finished" podID="2ff426c1-cd0d-465d-8dbf-4a8b2215d99d" containerID="1c618083b57acd80e74b7ac2e7ce8b5ae9dc508ac6137265063e03fde7d340f0" exitCode=0
Oct 30 03:24:46.383447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:46.360634    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq" event={"ID":"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d","Type":"ContainerDied","Data":"1c618083b57acd80e74b7ac2e7ce8b5ae9dc508ac6137265063e03fde7d340f0"}
Oct 30 03:24:46.811451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:46.810544    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 03:24:46.811451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:46.810611    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 03:24:47.129505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:47.128918    4278 patch_prober.go:29] interesting pod/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:24:47.129505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:47.128983    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 03:24:47.810151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:47.810119    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 03:24:47.811447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:47.810654    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 03:24:48.308451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:48.306531    4278 patch_prober.go:29] interesting pod/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:24:48.308451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:48.306594    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 03:24:48.535831 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:48.535789    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"]
Oct 30 03:24:48.536108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:48.536091    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"]
Oct 30 03:24:48.536463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:48.536447    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"]
Oct 30 03:24:48.837449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:48.826471    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 03:24:48.837449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:48.826534    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 03:24:48.837449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:48.833636    4278 generic.go:356] "Generic (PLEG): container finished" podID="c223f4da-5498-444d-923d-63e48082c902" containerID="bfe4bc1548fa5afe0a187b42ebae4f8a6da7824a3cf995bf9b2447807354c0d9" exitCode=0
Oct 30 03:24:48.888801 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:48.888759    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" event={"ID":"c223f4da-5498-444d-923d-63e48082c902","Type":"ContainerDied","Data":"bfe4bc1548fa5afe0a187b42ebae4f8a6da7824a3cf995bf9b2447807354c0d9"}
Oct 30 03:24:49.054846 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.054706    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:49.127454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.126675    4278 patch_prober.go:29] interesting pod/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:24:49.127454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.126732    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 03:24:49.141573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.140861    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-kube-api-access-ffg9b\" (UniqueName: \"kubernetes.io/projected/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-kube-api-access-ffg9b\") pod \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") "
Oct 30 03:24:49.141573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.140960    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-openshift-ca\") pod \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") "
Oct 30 03:24:49.141573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.141112    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-xks-ca\") pod \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\" (UID: \"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d\") "
Oct 30 03:24:49.148622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.148584    4278 generic.go:356] "Generic (PLEG): container finished" podID="d60341dc-c799-4814-8a04-d37ac0d30993" containerID="16cf572a8490a2d705d1598ab9fe2529a85b2b304d8c37997f738b80e3c65daf" exitCode=0
Oct 30 03:24:49.148892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.148814    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" event={"ID":"d60341dc-c799-4814-8a04-d37ac0d30993","Type":"ContainerDied","Data":"16cf572a8490a2d705d1598ab9fe2529a85b2b304d8c37997f738b80e3c65daf"}
Oct 30 03:24:49.149909 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.149867    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-openshift-ca" pod "2ff426c1-cd0d-465d-8dbf-4a8b2215d99d" (UID: "2ff426c1-cd0d-465d-8dbf-4a8b2215d99d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:49.184730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.184689    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 03:24:49.187845 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.187812    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-xks-ca" pod "2ff426c1-cd0d-465d-8dbf-4a8b2215d99d" (UID: "2ff426c1-cd0d-465d-8dbf-4a8b2215d99d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:49.193533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.189853    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-kube-api-access-ffg9b" pod "2ff426c1-cd0d-465d-8dbf-4a8b2215d99d" (UID: "2ff426c1-cd0d-465d-8dbf-4a8b2215d99d"). InnerVolumeSpecName "kube-api-access-ffg9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:49.197717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.196987    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:24:49.207703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.207679    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:24:49.320154 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.320114    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-socket\") pod \"c223f4da-5498-444d-923d-63e48082c902\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") "
Oct 30 03:24:49.320509 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.320487    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-credential-socket\") pod \"c223f4da-5498-444d-923d-63e48082c902\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") "
Oct 30 03:24:49.320742 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.320722    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-istio-token\" (UniqueName: \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-istio-token\") pod \"c223f4da-5498-444d-923d-63e48082c902\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") "
Oct 30 03:24:49.325171 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.325145    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c223f4da-5498-444d-923d-63e48082c902-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c223f4da-5498-444d-923d-63e48082c902-istiod-ca-cert\") pod \"c223f4da-5498-444d-923d-63e48082c902\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") "
Oct 30 03:24:49.325439 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.325403    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-data\") pod \"c223f4da-5498-444d-923d-63e48082c902\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") "
Oct 30 03:24:49.325709 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.325691    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-envoy\") pod \"c223f4da-5498-444d-923d-63e48082c902\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") "
Oct 30 03:24:49.325966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.325947    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-kube-api-access-sh2tw\" (UniqueName: \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-kube-api-access-sh2tw\") pod \"c223f4da-5498-444d-923d-63e48082c902\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") "
Oct 30 03:24:49.336905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.336827    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c223f4da-5498-444d-923d-63e48082c902-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c223f4da-5498-444d-923d-63e48082c902-istio-podinfo\") pod \"c223f4da-5498-444d-923d-63e48082c902\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") "
Oct 30 03:24:49.337278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.337255    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-certs\") pod \"c223f4da-5498-444d-923d-63e48082c902\" (UID: \"c223f4da-5498-444d-923d-63e48082c902\") "
Oct 30 03:24:49.348118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.338058    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffg9b\" (UniqueName: \"kubernetes.io/projected/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-kube-api-access-ffg9b\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.348483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.348459    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.348615 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.348591    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c223f4da-5498-444d-923d-63e48082c902-istiod-ca-cert" pod "c223f4da-5498-444d-923d-63e48082c902" (UID: "c223f4da-5498-444d-923d-63e48082c902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:49.348717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.348609    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.355703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.355677    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-data" pod "c223f4da-5498-444d-923d-63e48082c902" (UID: "c223f4da-5498-444d-923d-63e48082c902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:49.359754 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.359723    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-socket" pod "c223f4da-5498-444d-923d-63e48082c902" (UID: "c223f4da-5498-444d-923d-63e48082c902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:49.368293 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.368256    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-credential-socket" pod "c223f4da-5498-444d-923d-63e48082c902" (UID: "c223f4da-5498-444d-923d-63e48082c902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:49.396286 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.375610    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-envoy" pod "c223f4da-5498-444d-923d-63e48082c902" (UID: "c223f4da-5498-444d-923d-63e48082c902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:49.402517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.402480    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c223f4da-5498-444d-923d-63e48082c902-istio-podinfo" pod "c223f4da-5498-444d-923d-63e48082c902" (UID: "c223f4da-5498-444d-923d-63e48082c902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:49.423876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.423820    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-kube-api-access-sh2tw" pod "c223f4da-5498-444d-923d-63e48082c902" (UID: "c223f4da-5498-444d-923d-63e48082c902"). InnerVolumeSpecName "kube-api-access-sh2tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:49.424185 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.424145    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-certs" pod "c223f4da-5498-444d-923d-63e48082c902" (UID: "c223f4da-5498-444d-923d-63e48082c902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:49.438990 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.438949    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:24:49.442599 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.442547    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:24:49.456944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.456897    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-istio-token" pod "c223f4da-5498-444d-923d-63e48082c902" (UID: "c223f4da-5498-444d-923d-63e48082c902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:49.464526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.460722    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-socket\") pod \"d60341dc-c799-4814-8a04-d37ac0d30993\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") "
Oct 30 03:24:49.464526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.460810    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d60341dc-c799-4814-8a04-d37ac0d30993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60341dc-c799-4814-8a04-d37ac0d30993-istio-podinfo\") pod \"d60341dc-c799-4814-8a04-d37ac0d30993\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") "
Oct 30 03:24:49.464526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.460932    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-envoy\") pod \"d60341dc-c799-4814-8a04-d37ac0d30993\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") "
Oct 30 03:24:49.464526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.461015    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-data\") pod \"d60341dc-c799-4814-8a04-d37ac0d30993\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") "
Oct 30 03:24:49.464526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.461114    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-istio-token\" (UniqueName: \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-istio-token\") pod \"d60341dc-c799-4814-8a04-d37ac0d30993\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") "
Oct 30 03:24:49.464526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.461174    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-kube-api-access-p6nbq\" (UniqueName: \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-kube-api-access-p6nbq\") pod \"d60341dc-c799-4814-8a04-d37ac0d30993\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") "
Oct 30 03:24:49.464526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.461225    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-credential-socket\") pod \"d60341dc-c799-4814-8a04-d37ac0d30993\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") "
Oct 30 03:24:49.464526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.461269    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60341dc-c799-4814-8a04-d37ac0d30993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60341dc-c799-4814-8a04-d37ac0d30993-istiod-ca-cert\") pod \"d60341dc-c799-4814-8a04-d37ac0d30993\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") "
Oct 30 03:24:49.464526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.461326    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-certs\") pod \"d60341dc-c799-4814-8a04-d37ac0d30993\" (UID: \"d60341dc-c799-4814-8a04-d37ac0d30993\") "
Oct 30 03:24:49.473564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.473526    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-certs" pod "d60341dc-c799-4814-8a04-d37ac0d30993" (UID: "d60341dc-c799-4814-8a04-d37ac0d30993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:49.478746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.478716    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-istio-token" pod "d60341dc-c799-4814-8a04-d37ac0d30993" (UID: "d60341dc-c799-4814-8a04-d37ac0d30993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.481673    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-socket" pod "d60341dc-c799-4814-8a04-d37ac0d30993" (UID: "d60341dc-c799-4814-8a04-d37ac0d30993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.482280    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60341dc-c799-4814-8a04-d37ac0d30993-istiod-ca-cert" pod "d60341dc-c799-4814-8a04-d37ac0d30993" (UID: "d60341dc-c799-4814-8a04-d37ac0d30993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487217    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487246    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487260    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c223f4da-5498-444d-923d-63e48082c902-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487274    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487287    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487300    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh2tw\" (UniqueName: \"kubernetes.io/projected/c223f4da-5498-444d-923d-63e48082c902-kube-api-access-sh2tw\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487314    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487329    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c223f4da-5498-444d-923d-63e48082c902-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487343    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60341dc-c799-4814-8a04-d37ac0d30993-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487358    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487371    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487383    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.491800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.487396    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c223f4da-5498-444d-923d-63e48082c902-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.499459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.497673    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-credential-socket" pod "d60341dc-c799-4814-8a04-d37ac0d30993" (UID: "d60341dc-c799-4814-8a04-d37ac0d30993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:49.499459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.497929    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-data" pod "d60341dc-c799-4814-8a04-d37ac0d30993" (UID: "d60341dc-c799-4814-8a04-d37ac0d30993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:49.515697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.515664    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-envoy" pod "d60341dc-c799-4814-8a04-d37ac0d30993" (UID: "d60341dc-c799-4814-8a04-d37ac0d30993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:49.537805 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.537766    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-kube-api-access-p6nbq" pod "d60341dc-c799-4814-8a04-d37ac0d30993" (UID: "d60341dc-c799-4814-8a04-d37ac0d30993"). InnerVolumeSpecName "kube-api-access-p6nbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:49.556875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.556243    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d60341dc-c799-4814-8a04-d37ac0d30993-istio-podinfo" pod "d60341dc-c799-4814-8a04-d37ac0d30993" (UID: "d60341dc-c799-4814-8a04-d37ac0d30993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:49.609361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.601545    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.609361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.601586    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.609361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.601601    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6nbq\" (UniqueName: \"kubernetes.io/projected/d60341dc-c799-4814-8a04-d37ac0d30993-kube-api-access-p6nbq\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.609361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.601619    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60341dc-c799-4814-8a04-d37ac0d30993-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.609361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.601633    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60341dc-c799-4814-8a04-d37ac0d30993-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:49.812402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.812366    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 03:24:49.812658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:49.812630    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 03:24:50.126003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.125959    4278 patch_prober.go:29] interesting pod/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:24:50.126553 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.126043    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 03:24:50.171451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.167752    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"
Oct 30 03:24:50.181945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.181919    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"
Oct 30 03:24:50.188450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.185799    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"
Oct 30 03:24:50.203492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.170552    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq" event={"ID":"2ff426c1-cd0d-465d-8dbf-4a8b2215d99d","Type":"ContainerDied","Data":"49a5f81f537366f99ab5d28e83a93801f810b2db070992733e27b7c802ec240a"}
Oct 30 03:24:50.203492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.200592    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a5f81f537366f99ab5d28e83a93801f810b2db070992733e27b7c802ec240a"
Oct 30 03:24:50.226116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.225860    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn" event={"ID":"c223f4da-5498-444d-923d-63e48082c902","Type":"ContainerDied","Data":"afc24a72e9ec8f8a8e9e445ff4039099487d365deae146a3498856d1b61479da"}
Oct 30 03:24:50.226116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.225998    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c" event={"ID":"d60341dc-c799-4814-8a04-d37ac0d30993","Type":"ContainerDied","Data":"326afe2dfb13e5d67163b917f7198623fd7f6c49f50c7a6a04e148843e35730d"}
Oct 30 03:24:50.226116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.226099    4278 scope.go:122] "RemoveContainer" containerID="bfe4bc1548fa5afe0a187b42ebae4f8a6da7824a3cf995bf9b2447807354c0d9"
Oct 30 03:24:50.241579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.241554    4278 scope.go:122] "RemoveContainer" containerID="16cf572a8490a2d705d1598ab9fe2529a85b2b304d8c37997f738b80e3c65daf"
Oct 30 03:24:50.292481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.286219    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"]
Oct 30 03:24:50.305697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.305659    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-dfz2c"]
Oct 30 03:24:50.352825 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.352787    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"]
Oct 30 03:24:50.356824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.356796    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-t96bn"]
Oct 30 03:24:50.651757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.651728    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:24:50.651960 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.651787    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:24:50.812879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.812840    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 03:24:50.813650 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:50.813612    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 03:24:51.028760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:51.028729    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"]
Oct 30 03:24:51.032037 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:51.032010    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268cba7c-xd94d"]
Oct 30 03:24:51.129406 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:51.129379    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:24:51.233470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:51.233418    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:24:51.850282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:51.850246    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:24:51.852070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:51.852043    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:24:52.200284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:52.200199    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="518cf17e-ff4c-40c8-b88f-c98413289657" path="/var/lib/kubelet/pods/518cf17e-ff4c-40c8-b88f-c98413289657/volumes"
Oct 30 03:24:52.200885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:52.200863    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c223f4da-5498-444d-923d-63e48082c902" path="/var/lib/kubelet/pods/c223f4da-5498-444d-923d-63e48082c902/volumes"
Oct 30 03:24:52.201374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:52.201352    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" path="/var/lib/kubelet/pods/d60341dc-c799-4814-8a04-d37ac0d30993/volumes"
Oct 30 03:24:53.662325 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:53.662286    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"]
Oct 30 03:24:54.191698 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:24:54.191659    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:24:58.662802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:58.662728    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78" podUID="16051f97-ead7-430d-bbea-5da121098a02" containerName="payload-pre-processing" containerID="cri-o://ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d" gracePeriod=25
Oct 30 03:24:58.927950 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:58.927893    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 03:24:59.078947 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.078917    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16051f97-ead7-430d-bbea-5da121098a02-kube-api-access-vgz9r\" (UniqueName: \"kubernetes.io/projected/16051f97-ead7-430d-bbea-5da121098a02-kube-api-access-vgz9r\") pod \"16051f97-ead7-430d-bbea-5da121098a02\" (UID: \"16051f97-ead7-430d-bbea-5da121098a02\") "
Oct 30 03:24:59.079191 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.079177    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16051f97-ead7-430d-bbea-5da121098a02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16051f97-ead7-430d-bbea-5da121098a02-plugins-config-volume\") pod \"16051f97-ead7-430d-bbea-5da121098a02\" (UID: \"16051f97-ead7-430d-bbea-5da121098a02\") "
Oct 30 03:24:59.079522 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.079491    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16051f97-ead7-430d-bbea-5da121098a02-plugins-config-volume" pod "16051f97-ead7-430d-bbea-5da121098a02" (UID: "16051f97-ead7-430d-bbea-5da121098a02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:59.081013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.080992    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16051f97-ead7-430d-bbea-5da121098a02-kube-api-access-vgz9r" pod "16051f97-ead7-430d-bbea-5da121098a02" (UID: "16051f97-ead7-430d-bbea-5da121098a02"). InnerVolumeSpecName "kube-api-access-vgz9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:59.180573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.180493    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16051f97-ead7-430d-bbea-5da121098a02-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.180573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.180521    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgz9r\" (UniqueName: \"kubernetes.io/projected/16051f97-ead7-430d-bbea-5da121098a02-kube-api-access-vgz9r\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.291834 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.291802    4278 generic.go:356] "Generic (PLEG): container finished" podID="16051f97-ead7-430d-bbea-5da121098a02" containerID="ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d" exitCode=0
Oct 30 03:24:59.292001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.291853    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"
Oct 30 03:24:59.292001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.291889    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78" event={"ID":"16051f97-ead7-430d-bbea-5da121098a02","Type":"ContainerDied","Data":"ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d"}
Oct 30 03:24:59.292001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.291951    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78" event={"ID":"16051f97-ead7-430d-bbea-5da121098a02","Type":"ContainerDied","Data":"e428e1c8cd14fff71f99252fcf6d340ade751ac61bd70e7af94b6e6df6e8b6e5"}
Oct 30 03:24:59.292001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.291981    4278 scope.go:122] "RemoveContainer" containerID="ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d"
Oct 30 03:24:59.304302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.303753    4278 scope.go:122] "RemoveContainer" containerID="ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d"
Oct 30 03:24:59.306723 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:24:59.306694    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d\": container with ID starting with ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d not found: ID does not exist" containerID="ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d"
Oct 30 03:24:59.306834 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.306733    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d"} err="failed to get container status \"ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d\": rpc error: code = NotFound desc = could not find container \"ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d\": container with ID starting with ae73ba01dfea15282e1f2b3bdb286cfe21b78ce852057f25d9362e8021b7605d not found: ID does not exist"
Oct 30 03:24:59.350803 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.350776    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"]
Oct 30 03:24:59.363320 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:24:59.363297    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7d1788-85f96f7f55-bht78"]
Oct 30 03:25:00.199992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:00.199958    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16051f97-ead7-430d-bbea-5da121098a02" path="/var/lib/kubelet/pods/16051f97-ead7-430d-bbea-5da121098a02/volumes"
Oct 30 03:25:01.063580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:01.063537    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"]
Oct 30 03:25:01.074193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:01.074161    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdac2ce7-r6kgt"]
Oct 30 03:25:02.201864 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:02.201833    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6143d2-4417-4f00-97b0-cd796d221616" path="/var/lib/kubelet/pods/da6143d2-4417-4f00-97b0-cd796d221616/volumes"
Oct 30 03:25:04.182441 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.182363    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main" containerID="cri-o://90b34c0b60f6b8fd501f13c3dd59cc01b8d8a0e85e7a3aa6a00d9240e9e3ffc1" gracePeriod=45
Oct 30 03:25:04.216490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.216409    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main" containerID="cri-o://8016ea993532bb9669f3975d3fa55f3c8b27135c6b5acf932a6e72e970933b12" gracePeriod=45
Oct 30 03:25:04.230738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.230701    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" containerID="cri-o://beda273b6b62fc73fef3e98eae3f0692c98bc63dee3333c882d77d45bec2f36d" gracePeriod=45
Oct 30 03:25:04.258541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.258505    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 03:25:04.332887 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.332856    4278 generic.go:356] "Generic (PLEG): container finished" podID="a3ded0f0-18c0-4997-82b3-84111989074e" containerID="90b34c0b60f6b8fd501f13c3dd59cc01b8d8a0e85e7a3aa6a00d9240e9e3ffc1" exitCode=0
Oct 30 03:25:04.333037 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.332933    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" event={"ID":"a3ded0f0-18c0-4997-82b3-84111989074e","Type":"ContainerDied","Data":"90b34c0b60f6b8fd501f13c3dd59cc01b8d8a0e85e7a3aa6a00d9240e9e3ffc1"}
Oct 30 03:25:04.335211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.335089    4278 generic.go:356] "Generic (PLEG): container finished" podID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerID="8016ea993532bb9669f3975d3fa55f3c8b27135c6b5acf932a6e72e970933b12" exitCode=0
Oct 30 03:25:04.335211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.335174    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" event={"ID":"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2","Type":"ContainerDied","Data":"8016ea993532bb9669f3975d3fa55f3c8b27135c6b5acf932a6e72e970933b12"}
Oct 30 03:25:04.533956 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.533928    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 03:25:04.537743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.537705    4278 scope.go:122] "RemoveContainer" containerID="f7dfb1ed6979157de8801927be9be2000b2f334f3893ac9e542574397f8b875a"
Oct 30 03:25:04.548027 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.547998    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-kube-api-access-vv6sz\" (UniqueName: \"kubernetes.io/projected/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-kube-api-access-vv6sz\") pod \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") "
Oct 30 03:25:04.548200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.548046    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-home\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-home\") pod \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") "
Oct 30 03:25:04.548200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.548149    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-model-cache\") pod \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") "
Oct 30 03:25:04.548200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.548183    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-dshm\") pod \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") "
Oct 30 03:25:04.548472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.548301    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tls-certs\") pod \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") "
Oct 30 03:25:04.548472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.548419    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tmp-dir\") pod \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\" (UID: \"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2\") "
Oct 30 03:25:04.549307 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.549280    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-model-cache" pod "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" (UID: "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.549981 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.549705    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tmp-dir" pod "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" (UID: "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.549981 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.549964    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-home" pod "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" (UID: "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.552166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.552140    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-kube-api-access-vv6sz" pod "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" (UID: "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2"). InnerVolumeSpecName "kube-api-access-vv6sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:04.557810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.557737    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 03:25:04.558880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.558856    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tls-certs" pod "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" (UID: "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:04.558972 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.558948    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-dshm" pod "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" (UID: "544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.559116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.559095    4278 scope.go:122] "RemoveContainer" containerID="c2f3e7eaa178e48fd604632bbd5cfc2c69718e3c767e7905889e49506114db2e"
Oct 30 03:25:04.564396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.564369    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 03:25:04.570166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.570121    4278 scope.go:122] "RemoveContainer" containerID="a34998a7448928d6d06ed352c04a1e063a15e180ef9e40b2a52f0490539d5425"
Oct 30 03:25:04.586133 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.586107    4278 scope.go:122] "RemoveContainer" containerID="76cf74103bc08965cc9f201d37908e7bdff3f3b863fbe87eed443a56ed931623"
Oct 30 03:25:04.602079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.602048    4278 scope.go:122] "RemoveContainer" containerID="6e9863f26011b048cad576d856020723659a577fae0809fa877c08d06939c1a5"
Oct 30 03:25:04.613728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.613703    4278 scope.go:122] "RemoveContainer" containerID="f6f1d2affe7b4287fd40a080da9ef419189e9a1603cdf26790ab34cc96ff0cc5"
Oct 30 03:25:04.625841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.625819    4278 scope.go:122] "RemoveContainer" containerID="1715f8d551511144ab269a8ae32652c27314f7211a2fef3c0a65e78715dbca98"
Oct 30 03:25:04.638399 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.638367    4278 scope.go:122] "RemoveContainer" containerID="8016ea993532bb9669f3975d3fa55f3c8b27135c6b5acf932a6e72e970933b12"
Oct 30 03:25:04.649650 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.649617    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-dshm\") pod \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") "
Oct 30 03:25:04.649818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.649675    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-home\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-home\") pod \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") "
Oct 30 03:25:04.649818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.649725    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tmp-dir\") pod \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") "
Oct 30 03:25:04.649818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.649807    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ded0f0-18c0-4997-82b3-84111989074e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ded0f0-18c0-4997-82b3-84111989074e-tls-certs\") pod \"a3ded0f0-18c0-4997-82b3-84111989074e\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") "
Oct 30 03:25:04.650015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.649840    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-model-cache\") pod \"a3ded0f0-18c0-4997-82b3-84111989074e\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") "
Oct 30 03:25:04.650015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.649933    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-model-cache\") pod \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") "
Oct 30 03:25:04.650015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.649996    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tls-certs\") pod \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") "
Oct 30 03:25:04.650177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.650056    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-tmp-dir\") pod \"a3ded0f0-18c0-4997-82b3-84111989074e\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") "
Oct 30 03:25:04.650177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.650160    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-kube-api-access-tdvp9\" (UniqueName: \"kubernetes.io/projected/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-kube-api-access-tdvp9\") pod \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\" (UID: \"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e\") "
Oct 30 03:25:04.650289 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.650214    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ded0f0-18c0-4997-82b3-84111989074e-kube-api-access-nnplx\" (UniqueName: \"kubernetes.io/projected/a3ded0f0-18c0-4997-82b3-84111989074e-kube-api-access-nnplx\") pod \"a3ded0f0-18c0-4997-82b3-84111989074e\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") "
Oct 30 03:25:04.650289 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.650244    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-home\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-home\") pod \"a3ded0f0-18c0-4997-82b3-84111989074e\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") "
Oct 30 03:25:04.650413 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.650296    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-dshm\") pod \"a3ded0f0-18c0-4997-82b3-84111989074e\" (UID: \"a3ded0f0-18c0-4997-82b3-84111989074e\") "
Oct 30 03:25:04.650738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.649926    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-home" pod "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" (UID: "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.650738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.650296    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-model-cache" pod "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" (UID: "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.650738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.650488    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-model-cache" pod "a3ded0f0-18c0-4997-82b3-84111989074e" (UID: "a3ded0f0-18c0-4997-82b3-84111989074e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.650957 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.650710    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-tmp-dir" pod "a3ded0f0-18c0-4997-82b3-84111989074e" (UID: "a3ded0f0-18c0-4997-82b3-84111989074e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.651762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.651734    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-home" pod "a3ded0f0-18c0-4997-82b3-84111989074e" (UID: "a3ded0f0-18c0-4997-82b3-84111989074e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.651999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.651978    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652050    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652065    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652084    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv6sz\" (UniqueName: \"kubernetes.io/projected/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-kube-api-access-vv6sz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652098    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652112    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652125    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652138    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652153    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652164    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652176    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.652967 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652330    4278 scope.go:122] "RemoveContainer" containerID="beda273b6b62fc73fef3e98eae3f0692c98bc63dee3333c882d77d45bec2f36d"
Oct 30 03:25:04.652967 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.652874    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ded0f0-18c0-4997-82b3-84111989074e-tls-certs" pod "a3ded0f0-18c0-4997-82b3-84111989074e" (UID: "a3ded0f0-18c0-4997-82b3-84111989074e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:04.653658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.653602    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-kube-api-access-tdvp9" pod "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" (UID: "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e"). InnerVolumeSpecName "kube-api-access-tdvp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:04.654222 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.654197    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-dshm" pod "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" (UID: "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.654306 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.654249    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tmp-dir" pod "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" (UID: "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.654930 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.654909    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ded0f0-18c0-4997-82b3-84111989074e-kube-api-access-nnplx" pod "a3ded0f0-18c0-4997-82b3-84111989074e" (UID: "a3ded0f0-18c0-4997-82b3-84111989074e"). InnerVolumeSpecName "kube-api-access-nnplx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:04.655836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.655807    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-dshm" pod "a3ded0f0-18c0-4997-82b3-84111989074e" (UID: "a3ded0f0-18c0-4997-82b3-84111989074e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:04.657464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.657126    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tls-certs" pod "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" (UID: "d85dc7d6-1fcc-4f8a-836e-7079ab8c848e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:04.679659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.679632    4278 scope.go:122] "RemoveContainer" containerID="9d65d3d66c370d56e37bf2e93fcadddec9c0750b29a2288e76642239e36792e8"
Oct 30 03:25:04.692957 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.692933    4278 scope.go:122] "RemoveContainer" containerID="4307b81ed37caa4b3fe9e8ecfaa72e775d2b71ccfa1ab2c9a556a03fd5ce4574"
Oct 30 03:25:04.704370 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.704344    4278 scope.go:122] "RemoveContainer" containerID="90b34c0b60f6b8fd501f13c3dd59cc01b8d8a0e85e7a3aa6a00d9240e9e3ffc1"
Oct 30 03:25:04.753508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.753480    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.753619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.753510    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.753619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.753527    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ded0f0-18c0-4997-82b3-84111989074e-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.753619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.753538    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.753619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.753552    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdvp9\" (UniqueName: \"kubernetes.io/projected/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e-kube-api-access-tdvp9\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.753619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.753567    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnplx\" (UniqueName: \"kubernetes.io/projected/a3ded0f0-18c0-4997-82b3-84111989074e-kube-api-access-nnplx\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.753619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.753580    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ded0f0-18c0-4997-82b3-84111989074e-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.968526    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"]
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969653    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969677    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969693    4278 cpu_manager.go:414] "containerMap: removing container" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969703    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969716    4278 cpu_manager.go:414] "containerMap: removing container" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" containerName="istio-proxy"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969724    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" containerName="istio-proxy"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969753    4278 cpu_manager.go:414] "containerMap: removing container" podUID="16051f97-ead7-430d-bbea-5da121098a02" containerName="payload-pre-processing"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969763    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16051f97-ead7-430d-bbea-5da121098a02" containerName="payload-pre-processing"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969776    4278 cpu_manager.go:414] "containerMap: removing container" podUID="2ff426c1-cd0d-465d-8dbf-4a8b2215d99d" containerName="revoke-keys"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969784    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff426c1-cd0d-465d-8dbf-4a8b2215d99d" containerName="revoke-keys"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969811    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969820    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969856    4278 cpu_manager.go:414] "containerMap: removing container" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969862    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.969991    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" containerName="main"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.970007    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" containerName="main"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.970018    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" containerName="main"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.970025    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60341dc-c799-4814-8a04-d37ac0d30993" containerName="istio-proxy"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.970038    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="16051f97-ead7-430d-bbea-5da121098a02" containerName="payload-pre-processing"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.970046    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff426c1-cd0d-465d-8dbf-4a8b2215d99d" containerName="revoke-keys"
Oct 30 03:25:04.972505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.970057    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c223f4da-5498-444d-923d-63e48082c902" containerName="istio-proxy"
Oct 30 03:25:04.980548 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.976879    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:25:04.999449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:04.992756    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"]
Oct 30 03:25:05.003455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.000861    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-948965-dockercfg-9gvc4\""
Oct 30 03:25:05.003455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.001074    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-948965\""
Oct 30 03:25:05.060336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.060293    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbmz\" (UniqueName: \"kubernetes.io/projected/fae1d571-6e0f-48c0-b28a-e7caa6056527-kube-api-access-dbbmz\") pod \"payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf\" (UID: \"fae1d571-6e0f-48c0-b28a-e7caa6056527\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:25:05.060556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.060530    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae1d571-6e0f-48c0-b28a-e7caa6056527-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf\" (UID: \"fae1d571-6e0f-48c0-b28a-e7caa6056527\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:25:05.161408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.161368    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbmz\" (UniqueName: \"kubernetes.io/projected/fae1d571-6e0f-48c0-b28a-e7caa6056527-kube-api-access-dbbmz\") pod \"payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf\" (UID: \"fae1d571-6e0f-48c0-b28a-e7caa6056527\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:25:05.161621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.161543    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae1d571-6e0f-48c0-b28a-e7caa6056527-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf\" (UID: \"fae1d571-6e0f-48c0-b28a-e7caa6056527\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:25:05.162264 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.162239    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae1d571-6e0f-48c0-b28a-e7caa6056527-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf\" (UID: \"fae1d571-6e0f-48c0-b28a-e7caa6056527\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:25:05.183525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.183481    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbmz\" (UniqueName: \"kubernetes.io/projected/fae1d571-6e0f-48c0-b28a-e7caa6056527-kube-api-access-dbbmz\") pod \"payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf\" (UID: \"fae1d571-6e0f-48c0-b28a-e7caa6056527\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:25:05.316279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.316238    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:25:05.344252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.344209    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl" event={"ID":"a3ded0f0-18c0-4997-82b3-84111989074e","Type":"ContainerDied","Data":"36410c4dde804eeada623062d693fe570955814c0a77c90d7d3234cfaf466ec4"}
Oct 30 03:25:05.344464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.344269    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" event={"ID":"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e","Type":"ContainerDied","Data":"beda273b6b62fc73fef3e98eae3f0692c98bc63dee3333c882d77d45bec2f36d"}
Oct 30 03:25:05.344464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.344296    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw" event={"ID":"d85dc7d6-1fcc-4f8a-836e-7079ab8c848e","Type":"ContainerDied","Data":"d28e5a1d9071db8c505c558502fe4541c58028975f4d6925a1c1861b9fc0bafe"}
Oct 30 03:25:05.344464 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.344327    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg" event={"ID":"544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2","Type":"ContainerDied","Data":"7901b09258e29351659f375d1da864c5776fde8649e757adc5ba71ead3385224"}
Oct 30 03:25:05.344679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.344498    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"
Oct 30 03:25:05.344679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.344501    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"
Oct 30 03:25:05.350469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.345120    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"
Oct 30 03:25:05.477143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.477112    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"]
Oct 30 03:25:05.480297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.480268    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/simulator-w4-7d1788-kserve-58bc557594-zlwfl"]
Oct 30 03:25:05.513755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.513694    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"]
Oct 30 03:25:05.517166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.517141    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/premium-w4-7d1788-kserve-79ccc9d95-p96gw"]
Oct 30 03:25:05.579897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.579800    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"]
Oct 30 03:25:05.585964 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.585917    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-2-w4-7d1788-kserve-7bf8b64857-zbthg"]
Oct 30 03:25:05.620412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.620379    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"]
Oct 30 03:25:05.621074 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:25:05.621049    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae1d571_6e0f_48c0_b28a_e7caa6056527.slice/crio-4c8914488c3d4ef6502a78157e31d17127fc2d6838992670ab2d4d28113e5b60 WatchSource:0}: Error finding container 4c8914488c3d4ef6502a78157e31d17127fc2d6838992670ab2d4d28113e5b60: Status 404 returned error can't find the container with id 4c8914488c3d4ef6502a78157e31d17127fc2d6838992670ab2d4d28113e5b60
Oct 30 03:25:05.623229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:05.623213    4278 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:25:06.201498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:06.201459    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2" path="/var/lib/kubelet/pods/544a7f7e-ee3c-4e2c-9da6-aa486a4e40b2/volumes"
Oct 30 03:25:06.202060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:06.202043    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ded0f0-18c0-4997-82b3-84111989074e" path="/var/lib/kubelet/pods/a3ded0f0-18c0-4997-82b3-84111989074e/volumes"
Oct 30 03:25:06.202881 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:06.202547    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85dc7d6-1fcc-4f8a-836e-7079ab8c848e" path="/var/lib/kubelet/pods/d85dc7d6-1fcc-4f8a-836e-7079ab8c848e/volumes"
Oct 30 03:25:06.355859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:06.355819    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf" event={"ID":"fae1d571-6e0f-48c0-b28a-e7caa6056527","Type":"ContainerStarted","Data":"4c8914488c3d4ef6502a78157e31d17127fc2d6838992670ab2d4d28113e5b60"}
Oct 30 03:25:07.365323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:07.365281    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf" event={"ID":"fae1d571-6e0f-48c0-b28a-e7caa6056527","Type":"ContainerStarted","Data":"171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1"}
Oct 30 03:25:07.365873 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:07.365451    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:25:07.407696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:07.407637    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf" podStartSLOduration=2.222276088 podStartE2EDuration="3.407621793s" totalImagesPullingTime="1.185345705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:07.406668862 +0000 UTC m=+2645.330877050" watchObservedRunningTime="2026-10-30 03:25:07.407621793 +0000 UTC m=+2645.331829981"
Oct 30 03:25:09.016323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:09.016283    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:25:11.373798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:11.373765    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"]
Oct 30 03:25:11.385888 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:11.385859    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-086aee-ll2c5"]
Oct 30 03:25:12.222564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:12.222525    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03720fc6-d91b-40f9-9f4e-db5a047a04ec" path="/var/lib/kubelet/pods/03720fc6-d91b-40f9-9f4e-db5a047a04ec/volumes"
Oct 30 03:25:16.075655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:16.075551    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"]
Oct 30 03:25:16.087893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:16.087862    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35143558-c4x45"]
Oct 30 03:25:16.198705 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:16.198673    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0905bcd3-d19d-45e2-99a3-ca80d434452a" path="/var/lib/kubelet/pods/0905bcd3-d19d-45e2-99a3-ca80d434452a/volumes"
Oct 30 03:25:18.375246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.375218    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:25:18.774612 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.774581    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"]
Oct 30 03:25:18.778231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.778210    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:18.788920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.788894    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"]
Oct 30 03:25:18.791630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.791612    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:18.803216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.803193    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"]
Oct 30 03:25:18.811576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.811551    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:25:18.811741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.811551    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:25:18.822759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.822709    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"]
Oct 30 03:25:18.827915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.827890    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"]
Oct 30 03:25:18.828175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.828127    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf" podUID="75cd7123-3610-4420-adaf-bb6c62de7577" containerName="maas-api" containerID="cri-o://b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608" gracePeriod=30
Oct 30 03:25:18.961815 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.961783    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85tw\" (UniqueName: \"kubernetes.io/projected/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-kube-api-access-x85tw\") pod \"maas-api-revoke-sub-8c1dff0f-86r9j\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:18.962014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.961861    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lpcb\" (UniqueName: \"kubernetes.io/projected/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-kube-api-access-2lpcb\") pod \"maas-api-revoke-sub-486f9dbb-gwcxq\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:18.962014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.961899    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-486f9dbb-gwcxq\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:18.962014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.961933    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c1dff0f-86r9j\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:18.962014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.961950    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c1dff0f-86r9j\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:18.962014 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:18.961998    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-486f9dbb-gwcxq\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:19.063638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.063542    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-486f9dbb-gwcxq\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:19.063812 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.063729    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85tw\" (UniqueName: \"kubernetes.io/projected/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-kube-api-access-x85tw\") pod \"maas-api-revoke-sub-8c1dff0f-86r9j\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:19.063886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.063818    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lpcb\" (UniqueName: \"kubernetes.io/projected/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-kube-api-access-2lpcb\") pod \"maas-api-revoke-sub-486f9dbb-gwcxq\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:19.063886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.063868    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-486f9dbb-gwcxq\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:19.064007 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.063910    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c1dff0f-86r9j\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:19.064007 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.063935    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c1dff0f-86r9j\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:19.065258 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.064385    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-486f9dbb-gwcxq\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:19.065258 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.065188    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c1dff0f-86r9j\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:19.159319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.072984    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c1dff0f-86r9j\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:19.159319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.072986    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-486f9dbb-gwcxq\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:19.191465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.190381    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85tw\" (UniqueName: \"kubernetes.io/projected/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-kube-api-access-x85tw\") pod \"maas-api-revoke-sub-8c1dff0f-86r9j\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:19.200175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.199878    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lpcb\" (UniqueName: \"kubernetes.io/projected/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-kube-api-access-2lpcb\") pod \"maas-api-revoke-sub-486f9dbb-gwcxq\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:19.309662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.309326    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 03:25:19.392174 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.391730    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:19.408418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.408186    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:19.487549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.487507    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cd7123-3610-4420-adaf-bb6c62de7577-kube-api-access-n9mmk\" (UniqueName: \"kubernetes.io/projected/75cd7123-3610-4420-adaf-bb6c62de7577-kube-api-access-n9mmk\") pod \"75cd7123-3610-4420-adaf-bb6c62de7577\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") "
Oct 30 03:25:19.487733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.487595    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-metrics-certs\") pod \"75cd7123-3610-4420-adaf-bb6c62de7577\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") "
Oct 30 03:25:19.487733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.487654    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cd7123-3610-4420-adaf-bb6c62de7577-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cd7123-3610-4420-adaf-bb6c62de7577-openshift-service-ca\") pod \"75cd7123-3610-4420-adaf-bb6c62de7577\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") "
Oct 30 03:25:19.487733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.487718    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-maas-api-tls\") pod \"75cd7123-3610-4420-adaf-bb6c62de7577\" (UID: \"75cd7123-3610-4420-adaf-bb6c62de7577\") "
Oct 30 03:25:19.490844 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.490762    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cd7123-3610-4420-adaf-bb6c62de7577-openshift-service-ca" pod "75cd7123-3610-4420-adaf-bb6c62de7577" (UID: "75cd7123-3610-4420-adaf-bb6c62de7577"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:19.494480 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.494416    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-maas-api-tls" pod "75cd7123-3610-4420-adaf-bb6c62de7577" (UID: "75cd7123-3610-4420-adaf-bb6c62de7577"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:19.501606 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.501577    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-metrics-certs" pod "75cd7123-3610-4420-adaf-bb6c62de7577" (UID: "75cd7123-3610-4420-adaf-bb6c62de7577"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:19.509830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.509709    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cd7123-3610-4420-adaf-bb6c62de7577-kube-api-access-n9mmk" pod "75cd7123-3610-4420-adaf-bb6c62de7577" (UID: "75cd7123-3610-4420-adaf-bb6c62de7577"). InnerVolumeSpecName "kube-api-access-n9mmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:19.516862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.516837    4278 generic.go:356] "Generic (PLEG): container finished" podID="75cd7123-3610-4420-adaf-bb6c62de7577" containerID="b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608" exitCode=0
Oct 30 03:25:19.517094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.517076    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf" event={"ID":"75cd7123-3610-4420-adaf-bb6c62de7577","Type":"ContainerDied","Data":"b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608"}
Oct 30 03:25:19.517254 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.517233    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf" event={"ID":"75cd7123-3610-4420-adaf-bb6c62de7577","Type":"ContainerDied","Data":"8be42020782570642b100872fec76f495370cb3ebd6c640ff75f8817ee18a30b"}
Oct 30 03:25:19.517334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.517277    4278 scope.go:122] "RemoveContainer" containerID="b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608"
Oct 30 03:25:19.517556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.517539    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"
Oct 30 03:25:19.542340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.542316    4278 scope.go:122] "RemoveContainer" containerID="b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608"
Oct 30 03:25:19.542764 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:25:19.542725    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608\": container with ID starting with b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608 not found: ID does not exist" containerID="b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608"
Oct 30 03:25:19.542897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.542877    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608"} err="failed to get container status \"b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608\": rpc error: code = NotFound desc = could not find container \"b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608\": container with ID starting with b3145a482e87875169a820e3facf98ef6275ecaff2454d2a78e095cc492b8608 not found: ID does not exist"
Oct 30 03:25:19.589524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.589486    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-maas-api-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:19.589524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.589525    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9mmk\" (UniqueName: \"kubernetes.io/projected/75cd7123-3610-4420-adaf-bb6c62de7577-kube-api-access-n9mmk\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:19.589789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.589542    4278 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cd7123-3610-4420-adaf-bb6c62de7577-metrics-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:19.589789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.589556    4278 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cd7123-3610-4420-adaf-bb6c62de7577-openshift-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:19.589949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.589916    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"]
Oct 30 03:25:19.592968 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.592941    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-810119-7d895c8f88-hbchf"]
Oct 30 03:25:19.692899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.688308    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"]
Oct 30 03:25:19.696449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.693344    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-810119-xhrnz"]
Oct 30 03:25:19.725715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.725686    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"]
Oct 30 03:25:19.727555 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:25:19.727530    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37bc90c_c9c6_4983_b3d6_5ca30cd1f467.slice/crio-79df76a038828520b42be2762d2e16e2af8f99b357d2e402e14ec30156aac949 WatchSource:0}: Error finding container 79df76a038828520b42be2762d2e16e2af8f99b357d2e402e14ec30156aac949: Status 404 returned error can't find the container with id 79df76a038828520b42be2762d2e16e2af8f99b357d2e402e14ec30156aac949
Oct 30 03:25:19.738128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:19.738109    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"]
Oct 30 03:25:19.740770 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:25:19.740745    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7231391f_da96_46b1_ac3a_ee0f61fe1d3f.slice/crio-ced373e4a1ad23fe8fddec053bac648446708da0519d0e912f723f58621af3fe WatchSource:0}: Error finding container ced373e4a1ad23fe8fddec053bac648446708da0519d0e912f723f58621af3fe: Status 404 returned error can't find the container with id ced373e4a1ad23fe8fddec053bac648446708da0519d0e912f723f58621af3fe
Oct 30 03:25:20.199178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.199141    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307eda7e-9b6a-4ef2-9786-2b872dce22b1" path="/var/lib/kubelet/pods/307eda7e-9b6a-4ef2-9786-2b872dce22b1/volumes"
Oct 30 03:25:20.199702 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.199679    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75cd7123-3610-4420-adaf-bb6c62de7577" path="/var/lib/kubelet/pods/75cd7123-3610-4420-adaf-bb6c62de7577/volumes"
Oct 30 03:25:20.531547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.531443    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq" event={"ID":"7231391f-da96-46b1-ac3a-ee0f61fe1d3f","Type":"ContainerStarted","Data":"0517878a61770ee35dfcc1927c63e8da29ab4f94cbd246c52252c68e19a04297"}
Oct 30 03:25:20.531547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.531520    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq" event={"ID":"7231391f-da96-46b1-ac3a-ee0f61fe1d3f","Type":"ContainerStarted","Data":"ced373e4a1ad23fe8fddec053bac648446708da0519d0e912f723f58621af3fe"}
Oct 30 03:25:20.533367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.533338    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j" event={"ID":"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467","Type":"ContainerStarted","Data":"4dca9d636f58f3deb31b0205c85603fba97231689b905b0e845ee8013551c66a"}
Oct 30 03:25:20.533549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.533393    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j" event={"ID":"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467","Type":"ContainerStarted","Data":"79df76a038828520b42be2762d2e16e2af8f99b357d2e402e14ec30156aac949"}
Oct 30 03:25:20.589450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.582065    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq" podStartSLOduration=2.582043364 podStartE2EDuration="2.582043364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:20.581087522 +0000 UTC m=+2658.505295734" watchObservedRunningTime="2026-10-30 03:25:20.582043364 +0000 UTC m=+2658.506251554"
Oct 30 03:25:20.644392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.644312    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j" podStartSLOduration=2.64429198 podStartE2EDuration="2.64429198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:20.631229248 +0000 UTC m=+2658.555437436" watchObservedRunningTime="2026-10-30 03:25:20.64429198 +0000 UTC m=+2658.568500169"
Oct 30 03:25:20.920793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.920761    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"]
Oct 30 03:25:20.921310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.921297    4278 cpu_manager.go:414] "containerMap: removing container" podUID="75cd7123-3610-4420-adaf-bb6c62de7577" containerName="maas-api"
Oct 30 03:25:20.921353 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.921313    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cd7123-3610-4420-adaf-bb6c62de7577" containerName="maas-api"
Oct 30 03:25:20.921432 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.921413    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cd7123-3610-4420-adaf-bb6c62de7577" containerName="maas-api"
Oct 30 03:25:20.923739 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.923723    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:20.946558 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:20.946528    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"]
Oct 30 03:25:21.117624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.117527    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56g6n\" (UniqueName: \"kubernetes.io/projected/5abf941d-6159-4046-997b-3d47675f91c4-kube-api-access-56g6n\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:21.117624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.117577    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:21.117892 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.117870    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:21.218737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.218707    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:21.218931 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.218760    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56g6n\" (UniqueName: \"kubernetes.io/projected/5abf941d-6159-4046-997b-3d47675f91c4-kube-api-access-56g6n\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:21.218931 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.218790    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:21.219401 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.219376    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:21.221189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.221162    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:21.239648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.239626    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56g6n\" (UniqueName: \"kubernetes.io/projected/5abf941d-6159-4046-997b-3d47675f91c4-kube-api-access-56g6n\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:21.536157 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.536130    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:21.542955 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.542923    4278 generic.go:356] "Generic (PLEG): container finished" podID="7231391f-da96-46b1-ac3a-ee0f61fe1d3f" containerID="0517878a61770ee35dfcc1927c63e8da29ab4f94cbd246c52252c68e19a04297" exitCode=0
Oct 30 03:25:21.543106 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.543001    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq" event={"ID":"7231391f-da96-46b1-ac3a-ee0f61fe1d3f","Type":"ContainerDied","Data":"0517878a61770ee35dfcc1927c63e8da29ab4f94cbd246c52252c68e19a04297"}
Oct 30 03:25:21.545042 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.545016    4278 generic.go:356] "Generic (PLEG): container finished" podID="e37bc90c-c9c6-4983-b3d6-5ca30cd1f467" containerID="4dca9d636f58f3deb31b0205c85603fba97231689b905b0e845ee8013551c66a" exitCode=0
Oct 30 03:25:21.545159 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.545079    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j" event={"ID":"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467","Type":"ContainerDied","Data":"4dca9d636f58f3deb31b0205c85603fba97231689b905b0e845ee8013551c66a"}
Oct 30 03:25:21.840811 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:25:21.840783    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abf941d_6159_4046_997b_3d47675f91c4.slice/crio-af3c6ffde31cc7f6ce349e17ed6084809d76846adc0d7d3e80a0ff61775f568f WatchSource:0}: Error finding container af3c6ffde31cc7f6ce349e17ed6084809d76846adc0d7d3e80a0ff61775f568f: Status 404 returned error can't find the container with id af3c6ffde31cc7f6ce349e17ed6084809d76846adc0d7d3e80a0ff61775f568f
Oct 30 03:25:21.841895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:21.841858    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"]
Oct 30 03:25:22.552940 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.552903    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m" event={"ID":"5abf941d-6159-4046-997b-3d47675f91c4","Type":"ContainerStarted","Data":"4e10e3892882a626a8b5fa1ad4e689a096ab546e9afcf11cdbb447c581d973f8"}
Oct 30 03:25:22.553466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.552965    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m" event={"ID":"5abf941d-6159-4046-997b-3d47675f91c4","Type":"ContainerStarted","Data":"af3c6ffde31cc7f6ce349e17ed6084809d76846adc0d7d3e80a0ff61775f568f"}
Oct 30 03:25:22.712407 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.712386    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:22.736391 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.736369    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-kube-api-access-2lpcb\" (UniqueName: \"kubernetes.io/projected/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-kube-api-access-2lpcb\") pod \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") "
Oct 30 03:25:22.736515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.736453    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-openshift-ca\") pod \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") "
Oct 30 03:25:22.736515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.736507    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-xks-ca\") pod \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\" (UID: \"7231391f-da96-46b1-ac3a-ee0f61fe1d3f\") "
Oct 30 03:25:22.736852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.736816    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-openshift-ca" pod "7231391f-da96-46b1-ac3a-ee0f61fe1d3f" (UID: "7231391f-da96-46b1-ac3a-ee0f61fe1d3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:22.737095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.737075    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:22.738610 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.738587    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-xks-ca" pod "7231391f-da96-46b1-ac3a-ee0f61fe1d3f" (UID: "7231391f-da96-46b1-ac3a-ee0f61fe1d3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:22.738691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.738603    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-kube-api-access-2lpcb" pod "7231391f-da96-46b1-ac3a-ee0f61fe1d3f" (UID: "7231391f-da96-46b1-ac3a-ee0f61fe1d3f"). InnerVolumeSpecName "kube-api-access-2lpcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:22.753844 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.753821    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:22.755046 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.754991    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m" podStartSLOduration=2.754974897 podStartE2EDuration="2.754974897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:22.594830731 +0000 UTC m=+2660.519038944" watchObservedRunningTime="2026-10-30 03:25:22.754974897 +0000 UTC m=+2660.679183086"
Oct 30 03:25:22.837809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.837738    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-xks-ca\") pod \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") "
Oct 30 03:25:22.837984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.837824    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-openshift-ca\") pod \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") "
Oct 30 03:25:22.837984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.837855    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-kube-api-access-x85tw\" (UniqueName: \"kubernetes.io/projected/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-kube-api-access-x85tw\") pod \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\" (UID: \"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467\") "
Oct 30 03:25:22.838269 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.838235    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-openshift-ca" pod "e37bc90c-c9c6-4983-b3d6-5ca30cd1f467" (UID: "e37bc90c-c9c6-4983-b3d6-5ca30cd1f467"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:22.838269 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.838259    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lpcb\" (UniqueName: \"kubernetes.io/projected/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-kube-api-access-2lpcb\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:22.838467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.838273    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7231391f-da96-46b1-ac3a-ee0f61fe1d3f-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:22.839939 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.839915    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-kube-api-access-x85tw" pod "e37bc90c-c9c6-4983-b3d6-5ca30cd1f467" (UID: "e37bc90c-c9c6-4983-b3d6-5ca30cd1f467"). InnerVolumeSpecName "kube-api-access-x85tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:22.840033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.839962    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-xks-ca" pod "e37bc90c-c9c6-4983-b3d6-5ca30cd1f467" (UID: "e37bc90c-c9c6-4983-b3d6-5ca30cd1f467"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:22.939552 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.939516    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:22.939552 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.939546    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x85tw\" (UniqueName: \"kubernetes.io/projected/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-kube-api-access-x85tw\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:22.939552 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:22.939557    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:23.561810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:23.561782    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"
Oct 30 03:25:23.561810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:23.561798    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq" event={"ID":"7231391f-da96-46b1-ac3a-ee0f61fe1d3f","Type":"ContainerDied","Data":"ced373e4a1ad23fe8fddec053bac648446708da0519d0e912f723f58621af3fe"}
Oct 30 03:25:23.562285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:23.561844    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ced373e4a1ad23fe8fddec053bac648446708da0519d0e912f723f58621af3fe"
Oct 30 03:25:23.563195 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:23.563174    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"
Oct 30 03:25:23.563336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:23.563209    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j" event={"ID":"e37bc90c-c9c6-4983-b3d6-5ca30cd1f467","Type":"ContainerDied","Data":"79df76a038828520b42be2762d2e16e2af8f99b357d2e402e14ec30156aac949"}
Oct 30 03:25:23.563336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:23.563246    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79df76a038828520b42be2762d2e16e2af8f99b357d2e402e14ec30156aac949"
Oct 30 03:25:23.565126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:23.565101    4278 generic.go:356] "Generic (PLEG): container finished" podID="5abf941d-6159-4046-997b-3d47675f91c4" containerID="4e10e3892882a626a8b5fa1ad4e689a096ab546e9afcf11cdbb447c581d973f8" exitCode=0
Oct 30 03:25:23.565230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:23.565185    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m" event={"ID":"5abf941d-6159-4046-997b-3d47675f91c4","Type":"ContainerDied","Data":"4e10e3892882a626a8b5fa1ad4e689a096ab546e9afcf11cdbb447c581d973f8"}
Oct 30 03:25:24.190921 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:25:24.190887    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:25:24.763615 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:24.763587    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:24.864143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:24.864066    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abf941d-6159-4046-997b-3d47675f91c4-kube-api-access-56g6n\" (UniqueName: \"kubernetes.io/projected/5abf941d-6159-4046-997b-3d47675f91c4-kube-api-access-56g6n\") pod \"5abf941d-6159-4046-997b-3d47675f91c4\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") "
Oct 30 03:25:24.864143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:24.864117    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-openshift-ca\") pod \"5abf941d-6159-4046-997b-3d47675f91c4\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") "
Oct 30 03:25:24.864348 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:24.864303    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-xks-ca\") pod \"5abf941d-6159-4046-997b-3d47675f91c4\" (UID: \"5abf941d-6159-4046-997b-3d47675f91c4\") "
Oct 30 03:25:24.864537 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:24.864507    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-openshift-ca" pod "5abf941d-6159-4046-997b-3d47675f91c4" (UID: "5abf941d-6159-4046-997b-3d47675f91c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:24.864954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:24.864927    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:24.866358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:24.866333    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abf941d-6159-4046-997b-3d47675f91c4-kube-api-access-56g6n" pod "5abf941d-6159-4046-997b-3d47675f91c4" (UID: "5abf941d-6159-4046-997b-3d47675f91c4"). InnerVolumeSpecName "kube-api-access-56g6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:24.866525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:24.866465    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-xks-ca" pod "5abf941d-6159-4046-997b-3d47675f91c4" (UID: "5abf941d-6159-4046-997b-3d47675f91c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:24.965814 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:24.965783    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abf941d-6159-4046-997b-3d47675f91c4-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:24.965814 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:24.965811    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56g6n\" (UniqueName: \"kubernetes.io/projected/5abf941d-6159-4046-997b-3d47675f91c4-kube-api-access-56g6n\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:25.583501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:25.583460    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"
Oct 30 03:25:25.583707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:25.583464    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m" event={"ID":"5abf941d-6159-4046-997b-3d47675f91c4","Type":"ContainerDied","Data":"af3c6ffde31cc7f6ce349e17ed6084809d76846adc0d7d3e80a0ff61775f568f"}
Oct 30 03:25:25.583707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:25.583604    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af3c6ffde31cc7f6ce349e17ed6084809d76846adc0d7d3e80a0ff61775f568f"
Oct 30 03:25:27.397358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.397313    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"]
Oct 30 03:25:27.405544 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.405488    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy" containerID="cri-o://2dfe3fac9ff05338e0ebfcd7c22a55c7c6c6ecbcbc38b599122fb13095dfa069" gracePeriod=30
Oct 30 03:25:27.970630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.970590    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"]
Oct 30 03:25:27.971277 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.971255    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5abf941d-6159-4046-997b-3d47675f91c4" containerName="revoke-keys"
Oct 30 03:25:27.971277 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.971277    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abf941d-6159-4046-997b-3d47675f91c4" containerName="revoke-keys"
Oct 30 03:25:27.971534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.971297    4278 cpu_manager.go:414] "containerMap: removing container" podUID="7231391f-da96-46b1-ac3a-ee0f61fe1d3f" containerName="revoke-keys"
Oct 30 03:25:27.971534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.971306    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7231391f-da96-46b1-ac3a-ee0f61fe1d3f" containerName="revoke-keys"
Oct 30 03:25:27.971534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.971335    4278 cpu_manager.go:414] "containerMap: removing container" podUID="e37bc90c-c9c6-4983-b3d6-5ca30cd1f467" containerName="revoke-keys"
Oct 30 03:25:27.971534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.971343    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37bc90c-c9c6-4983-b3d6-5ca30cd1f467" containerName="revoke-keys"
Oct 30 03:25:27.971750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.971554    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abf941d-6159-4046-997b-3d47675f91c4" containerName="revoke-keys"
Oct 30 03:25:27.971750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.971572    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="7231391f-da96-46b1-ac3a-ee0f61fe1d3f" containerName="revoke-keys"
Oct 30 03:25:27.971750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.971586    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37bc90c-c9c6-4983-b3d6-5ca30cd1f467" containerName="revoke-keys"
Oct 30 03:25:27.974182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.974161    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:27.989348 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:27.989328    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e9528f-serving-cert\""
Oct 30 03:25:28.007383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.007357    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"]
Oct 30 03:25:28.014243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.014216    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbtr\" (UniqueName: \"kubernetes.io/projected/f07c99c1-1acb-46cd-be80-25c6ba04dae3-kube-api-access-pcbtr\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.014692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.014407    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-metrics-certs\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.014692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.014452    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07c99c1-1acb-46cd-be80-25c6ba04dae3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.014692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.014493    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.115368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.115331    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-metrics-certs\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.115566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.115368    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07c99c1-1acb-46cd-be80-25c6ba04dae3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.115566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.115414    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.115690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.115589    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbtr\" (UniqueName: \"kubernetes.io/projected/f07c99c1-1acb-46cd-be80-25c6ba04dae3-kube-api-access-pcbtr\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.115690 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:25:28.115620    4278 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-serving-cert: secret "maas-api-e2e-ipp-a-e9528f-serving-cert" not found
Oct 30 03:25:28.115791 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:25:28.115709    4278 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-maas-api-tls podName:f07c99c1-1acb-46cd-be80-25c6ba04dae3 nodeName:}" failed. No retries permitted until 2026-10-30 03:25:28.61568584 +0000 UTC m=+2666.539894011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-maas-api-tls") pod "maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk" (UID: "f07c99c1-1acb-46cd-be80-25c6ba04dae3") : secret "maas-api-e2e-ipp-a-e9528f-serving-cert" not found
Oct 30 03:25:28.116382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.116354    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07c99c1-1acb-46cd-be80-25c6ba04dae3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.117814 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.117790    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-metrics-certs\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.136405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.136380    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbtr\" (UniqueName: \"kubernetes.io/projected/f07c99c1-1acb-46cd-be80-25c6ba04dae3-kube-api-access-pcbtr\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.202606 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.202571    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"]
Oct 30 03:25:28.205473 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.205453    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.224301 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.224204    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:25:28.224417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.224364    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:25:28.228234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.228205    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"]
Oct 30 03:25:28.320740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.320189    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5d9\" (UniqueName: \"kubernetes.io/projected/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-kube-api-access-qp5d9\") pod \"maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.320740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.320289    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.320740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.320396    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.422304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.422269    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5d9\" (UniqueName: \"kubernetes.io/projected/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-kube-api-access-qp5d9\") pod \"maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.422806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.422332    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.422806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.422382    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.423152 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.423130    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.424828 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.424808    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.443965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.443938    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5d9\" (UniqueName: \"kubernetes.io/projected/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-kube-api-access-qp5d9\") pod \"maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.518090 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.518066    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:28.625366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.625328    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.627757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.627732    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:28.820826 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.820795    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"]
Oct 30 03:25:28.822681 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:25:28.822653    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09c888d_67bc_42b4_adb4_fc8ec1a9924b.slice/crio-71f30558d811f2e3060cc3f25112d5628d9390bdff8f99b43be5ac2915ada1f5 WatchSource:0}: Error finding container 71f30558d811f2e3060cc3f25112d5628d9390bdff8f99b43be5ac2915ada1f5: Status 404 returned error can't find the container with id 71f30558d811f2e3060cc3f25112d5628d9390bdff8f99b43be5ac2915ada1f5
Oct 30 03:25:28.893757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:28.886742    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:29.211472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:29.211446    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"]
Oct 30 03:25:29.212556 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:25:29.212525    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07c99c1_1acb_46cd_be80_25c6ba04dae3.slice/crio-395816f7313e4d4005beccb951563593f67070281ef1a7a14837f724718639d1 WatchSource:0}: Error finding container 395816f7313e4d4005beccb951563593f67070281ef1a7a14837f724718639d1: Status 404 returned error can't find the container with id 395816f7313e4d4005beccb951563593f67070281ef1a7a14837f724718639d1
Oct 30 03:25:29.616477 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:29.616438    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk" event={"ID":"f07c99c1-1acb-46cd-be80-25c6ba04dae3","Type":"ContainerStarted","Data":"d4abcab6e68e4289d240281dcf2571e878465b12d46119d65bb5d3e1aeb85e57"}
Oct 30 03:25:29.616943 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:29.616502    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk" event={"ID":"f07c99c1-1acb-46cd-be80-25c6ba04dae3","Type":"ContainerStarted","Data":"395816f7313e4d4005beccb951563593f67070281ef1a7a14837f724718639d1"}
Oct 30 03:25:29.616943 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:29.616660    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:29.618258 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:29.618232    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl" event={"ID":"a09c888d-67bc-42b4-adb4-fc8ec1a9924b","Type":"ContainerStarted","Data":"36aa24facc3ce83b47b10e7ad6e93fa3a32b4d10c1e1db5880150fe1ee701eb8"}
Oct 30 03:25:29.618381 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:29.618278    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl" event={"ID":"a09c888d-67bc-42b4-adb4-fc8ec1a9924b","Type":"ContainerStarted","Data":"71f30558d811f2e3060cc3f25112d5628d9390bdff8f99b43be5ac2915ada1f5"}
Oct 30 03:25:29.662465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:29.662344    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk" podStartSLOduration=2.376211921 podStartE2EDuration="2.662327361s" totalImagesPullingTime="286.11544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:29.658418217 +0000 UTC m=+2667.582626415" watchObservedRunningTime="2026-10-30 03:25:29.662327361 +0000 UTC m=+2667.586535549"
Oct 30 03:25:29.699847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:29.699792    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl" podStartSLOduration=1.6997788809999999 podStartE2EDuration="1.699778881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:29.698519679 +0000 UTC m=+2667.622727869" watchObservedRunningTime="2026-10-30 03:25:29.699778881 +0000 UTC m=+2667.623987069"
Oct 30 03:25:30.627155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:30.627116    4278 generic.go:356] "Generic (PLEG): container finished" podID="a09c888d-67bc-42b4-adb4-fc8ec1a9924b" containerID="36aa24facc3ce83b47b10e7ad6e93fa3a32b4d10c1e1db5880150fe1ee701eb8" exitCode=0
Oct 30 03:25:30.627592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:30.627208    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl" event={"ID":"a09c888d-67bc-42b4-adb4-fc8ec1a9924b","Type":"ContainerDied","Data":"36aa24facc3ce83b47b10e7ad6e93fa3a32b4d10c1e1db5880150fe1ee701eb8"}
Oct 30 03:25:31.032007 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.031970    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"]
Oct 30 03:25:31.035518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.035481    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3406363b-dwhh7"]
Oct 30 03:25:31.794983 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.794960    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:31.874328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.874295    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-openshift-ca\") pod \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") "
Oct 30 03:25:31.874555 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.874376    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-xks-ca\") pod \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") "
Oct 30 03:25:31.874632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.874567    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-kube-api-access-qp5d9\" (UniqueName: \"kubernetes.io/projected/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-kube-api-access-qp5d9\") pod \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\" (UID: \"a09c888d-67bc-42b4-adb4-fc8ec1a9924b\") "
Oct 30 03:25:31.874811 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.874758    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-openshift-ca" pod "a09c888d-67bc-42b4-adb4-fc8ec1a9924b" (UID: "a09c888d-67bc-42b4-adb4-fc8ec1a9924b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:31.875167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.875138    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:31.876893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.876862    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-xks-ca" pod "a09c888d-67bc-42b4-adb4-fc8ec1a9924b" (UID: "a09c888d-67bc-42b4-adb4-fc8ec1a9924b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:31.877161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.877131    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-kube-api-access-qp5d9" pod "a09c888d-67bc-42b4-adb4-fc8ec1a9924b" (UID: "a09c888d-67bc-42b4-adb4-fc8ec1a9924b"). InnerVolumeSpecName "kube-api-access-qp5d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:31.976168 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.976128    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:31.976168 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:31.976159    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp5d9\" (UniqueName: \"kubernetes.io/projected/a09c888d-67bc-42b4-adb4-fc8ec1a9924b-kube-api-access-qp5d9\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.200201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.200121    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bafe140-0707-40d7-ac96-302a8d01be97" path="/var/lib/kubelet/pods/0bafe140-0707-40d7-ac96-302a8d01be97/volumes"
Oct 30 03:25:32.644886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.644850    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"
Oct 30 03:25:32.645059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.644930    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl" event={"ID":"a09c888d-67bc-42b4-adb4-fc8ec1a9924b","Type":"ContainerDied","Data":"71f30558d811f2e3060cc3f25112d5628d9390bdff8f99b43be5ac2915ada1f5"}
Oct 30 03:25:32.645059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.644983    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f30558d811f2e3060cc3f25112d5628d9390bdff8f99b43be5ac2915ada1f5"
Oct 30 03:25:32.647773 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.647749    4278 generic.go:356] "Generic (PLEG): container finished" podID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerID="2dfe3fac9ff05338e0ebfcd7c22a55c7c6c6ecbcbc38b599122fb13095dfa069" exitCode=0
Oct 30 03:25:32.647945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.647810    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" event={"ID":"ff3e5048-ada2-4a8e-bcff-66e9c831fd00","Type":"ContainerDied","Data":"2dfe3fac9ff05338e0ebfcd7c22a55c7c6c6ecbcbc38b599122fb13095dfa069"}
Oct 30 03:25:32.675862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.675842    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 03:25:32.805211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805142    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-kube-api-access-5jjv4\" (UniqueName: \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-kube-api-access-5jjv4\") pod \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") "
Oct 30 03:25:32.805211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805197    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-podinfo\") pod \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") "
Oct 30 03:25:32.805853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805263    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istiod-ca-cert\") pod \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") "
Oct 30 03:25:32.805853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805369    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-token\" (UniqueName: \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-token\") pod \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") "
Oct 30 03:25:32.805853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805476    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-envoy\") pod \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") "
Oct 30 03:25:32.805853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805522    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-certs\") pod \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") "
Oct 30 03:25:32.805853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805585    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-data\") pod \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") "
Oct 30 03:25:32.805853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805646    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-credential-socket\") pod \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") "
Oct 30 03:25:32.805853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805704    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-socket\") pod \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\" (UID: \"ff3e5048-ada2-4a8e-bcff-66e9c831fd00\") "
Oct 30 03:25:32.805853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805795    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istiod-ca-cert" pod "ff3e5048-ada2-4a8e-bcff-66e9c831fd00" (UID: "ff3e5048-ada2-4a8e-bcff-66e9c831fd00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:32.806261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.805996    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-data" pod "ff3e5048-ada2-4a8e-bcff-66e9c831fd00" (UID: "ff3e5048-ada2-4a8e-bcff-66e9c831fd00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:32.806261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.806094    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-credential-socket" pod "ff3e5048-ada2-4a8e-bcff-66e9c831fd00" (UID: "ff3e5048-ada2-4a8e-bcff-66e9c831fd00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:32.806261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.806121    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-certs" pod "ff3e5048-ada2-4a8e-bcff-66e9c831fd00" (UID: "ff3e5048-ada2-4a8e-bcff-66e9c831fd00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:32.806261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.806220    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-socket" pod "ff3e5048-ada2-4a8e-bcff-66e9c831fd00" (UID: "ff3e5048-ada2-4a8e-bcff-66e9c831fd00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:32.806545 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.806520    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.806649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.806551    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.806649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.806566    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.806649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.806579    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.806649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.806594    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.807903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.807876    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-podinfo" pod "ff3e5048-ada2-4a8e-bcff-66e9c831fd00" (UID: "ff3e5048-ada2-4a8e-bcff-66e9c831fd00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:25:32.808029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.807942    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-kube-api-access-5jjv4" pod "ff3e5048-ada2-4a8e-bcff-66e9c831fd00" (UID: "ff3e5048-ada2-4a8e-bcff-66e9c831fd00"). InnerVolumeSpecName "kube-api-access-5jjv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:32.808029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.808000    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-token" pod "ff3e5048-ada2-4a8e-bcff-66e9c831fd00" (UID: "ff3e5048-ada2-4a8e-bcff-66e9c831fd00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:32.808371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.808355    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-envoy" pod "ff3e5048-ada2-4a8e-bcff-66e9c831fd00" (UID: "ff3e5048-ada2-4a8e-bcff-66e9c831fd00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:32.908154 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.908124    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.908154 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.908150    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.908154 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.908162    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jjv4\" (UniqueName: \"kubernetes.io/projected/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-kube-api-access-5jjv4\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.908384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:32.908172    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3e5048-ada2-4a8e-bcff-66e9c831fd00-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:33.657507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:33.657468    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"
Oct 30 03:25:33.657507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:33.657488    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z" event={"ID":"ff3e5048-ada2-4a8e-bcff-66e9c831fd00","Type":"ContainerDied","Data":"b84e4f56d02af4f169b7fb5065b63a982ef744df7e61c23273493cbc6b2399d2"}
Oct 30 03:25:33.657778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:33.657568    4278 scope.go:122] "RemoveContainer" containerID="2dfe3fac9ff05338e0ebfcd7c22a55c7c6c6ecbcbc38b599122fb13095dfa069"
Oct 30 03:25:33.728438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:33.728398    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"]
Oct 30 03:25:33.741977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:33.741946    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-5ml9z"]
Oct 30 03:25:34.200763 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:34.200728    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" path="/var/lib/kubelet/pods/ff3e5048-ada2-4a8e-bcff-66e9c831fd00/volumes"
Oct 30 03:25:35.265707 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.265676    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"]
Oct 30 03:25:35.266299 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.266273    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy"
Oct 30 03:25:35.266299 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.266292    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy"
Oct 30 03:25:35.266500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.266312    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a09c888d-67bc-42b4-adb4-fc8ec1a9924b" containerName="revoke-keys"
Oct 30 03:25:35.266500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.266317    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09c888d-67bc-42b4-adb4-fc8ec1a9924b" containerName="revoke-keys"
Oct 30 03:25:35.266500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.266397    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09c888d-67bc-42b4-adb4-fc8ec1a9924b" containerName="revoke-keys"
Oct 30 03:25:35.266500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.266415    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3e5048-ada2-4a8e-bcff-66e9c831fd00" containerName="istio-proxy"
Oct 30 03:25:35.269847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.269830    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.297793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.297761    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-874ab5-openshift-default-dockercfg-kmjhl\""
Oct 30 03:25:35.335448 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.329725    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"]
Oct 30 03:25:35.347332 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.347291    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-token\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.347524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.347465    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-data\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.347655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.347629    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-credential-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.347715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.347696    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-envoy\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.347759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.347746    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-certs\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.347795 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.347777    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-podinfo\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.347895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.347879    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksgt\" (UniqueName: \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-kube-api-access-hksgt\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.347928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.347913    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.347982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.347967    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istiod-ca-cert\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449292    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istiod-ca-cert\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449342    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-token\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449386    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-data\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449476    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-credential-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449506    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-envoy\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449532    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-certs\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449653    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-podinfo\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449763    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksgt\" (UniqueName: \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-kube-api-access-hksgt\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449790    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449877    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-data\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.449975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449908    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-credential-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.450087 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.449989    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-certs\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.450148 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.450092    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.450148 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.450128    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istiod-ca-cert\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.451725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.451703    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-envoy\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.452185 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.452164    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-podinfo\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.472244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.472217    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-token\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.472352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.472244    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksgt\" (UniqueName: \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-kube-api-access-hksgt\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.583205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.583133    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:35.633445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.633406    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:25:35.899965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.899935    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"]
Oct 30 03:25:35.901048 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:25:35.901018    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e25ae83_0654_4360_a8a0_f902f1bd67ad.slice/crio-870bc346ac2465df1654df0aa0274894854f16d20ce24b5e3bdd6c2f155fe26d WatchSource:0}: Error finding container 870bc346ac2465df1654df0aa0274894854f16d20ce24b5e3bdd6c2f155fe26d: Status 404 returned error can't find the container with id 870bc346ac2465df1654df0aa0274894854f16d20ce24b5e3bdd6c2f155fe26d
Oct 30 03:25:35.904273 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.904240    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:25:35.904359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.904309    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:25:35.904359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:35.904339    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:25:36.693500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:36.693462    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" event={"ID":"5e25ae83-0654-4360-a8a0-f902f1bd67ad","Type":"ContainerStarted","Data":"d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c"}
Oct 30 03:25:36.693995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:36.693528    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" event={"ID":"5e25ae83-0654-4360-a8a0-f902f1bd67ad","Type":"ContainerStarted","Data":"870bc346ac2465df1654df0aa0274894854f16d20ce24b5e3bdd6c2f155fe26d"}
Oct 30 03:25:36.785780 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:36.785730    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" podStartSLOduration=1.785712044 podStartE2EDuration="1.785712044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:36.749202677 +0000 UTC m=+2674.673410865" watchObservedRunningTime="2026-10-30 03:25:36.785712044 +0000 UTC m=+2674.709920233"
Oct 30 03:25:37.583772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:37.583731    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:37.585553 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:37.585523    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 30 03:25:37.585737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:37.585581    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" podUID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 30 03:25:38.583895 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:38.583858    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 30 03:25:38.584726 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:38.584690    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" podUID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 30 03:25:39.583855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:39.583804    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 30 03:25:39.584059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:39.583871    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" podUID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 30 03:25:40.584027 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:40.583974    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 30 03:25:40.584245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:40.584047    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" podUID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 30 03:25:41.583850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:41.583808    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 30 03:25:41.584162 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:41.584127    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" podUID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 30 03:25:42.589068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:42.589036    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:42.589539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:42.589520    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:42.590309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:42.590283    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:25:50.559522 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:50.552259    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:25:54.190583 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:25:54.190547    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:25:56.552304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:56.552265    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"]
Oct 30 03:25:56.552824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:56.552316    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7d1788-t6znq"]
Oct 30 03:25:58.200575 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.200543    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff426c1-cd0d-465d-8dbf-4a8b2215d99d" path="/var/lib/kubelet/pods/2ff426c1-cd0d-465d-8dbf-4a8b2215d99d/volumes"
Oct 30 03:25:58.254023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.253991    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"]
Oct 30 03:25:58.254205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.254184    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6" podUID="e873cfdf-044e-41a6-86dc-14f8c4724748" containerName="maas-api" containerID="cri-o://973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff" gracePeriod=30
Oct 30 03:25:58.596542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.596511    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:25:58.680615 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.680572    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e873cfdf-044e-41a6-86dc-14f8c4724748-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e873cfdf-044e-41a6-86dc-14f8c4724748-openshift-service-ca\") pod \"e873cfdf-044e-41a6-86dc-14f8c4724748\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") "
Oct 30 03:25:58.680810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.680757    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-maas-api-tls\") pod \"e873cfdf-044e-41a6-86dc-14f8c4724748\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") "
Oct 30 03:25:58.680884 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.680853    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-metrics-certs\") pod \"e873cfdf-044e-41a6-86dc-14f8c4724748\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") "
Oct 30 03:25:58.680947 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.680910    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e873cfdf-044e-41a6-86dc-14f8c4724748-kube-api-access-w7cm7\" (UniqueName: \"kubernetes.io/projected/e873cfdf-044e-41a6-86dc-14f8c4724748-kube-api-access-w7cm7\") pod \"e873cfdf-044e-41a6-86dc-14f8c4724748\" (UID: \"e873cfdf-044e-41a6-86dc-14f8c4724748\") "
Oct 30 03:25:58.681360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.681310    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e873cfdf-044e-41a6-86dc-14f8c4724748-openshift-service-ca" pod "e873cfdf-044e-41a6-86dc-14f8c4724748" (UID: "e873cfdf-044e-41a6-86dc-14f8c4724748"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:58.681531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.681412    4278 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e873cfdf-044e-41a6-86dc-14f8c4724748-openshift-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:58.683855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.683815    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-metrics-certs" pod "e873cfdf-044e-41a6-86dc-14f8c4724748" (UID: "e873cfdf-044e-41a6-86dc-14f8c4724748"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:58.684236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.684205    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e873cfdf-044e-41a6-86dc-14f8c4724748-kube-api-access-w7cm7" pod "e873cfdf-044e-41a6-86dc-14f8c4724748" (UID: "e873cfdf-044e-41a6-86dc-14f8c4724748"). InnerVolumeSpecName "kube-api-access-w7cm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:58.684538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.684504    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-maas-api-tls" pod "e873cfdf-044e-41a6-86dc-14f8c4724748" (UID: "e873cfdf-044e-41a6-86dc-14f8c4724748"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:58.782367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.782329    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-maas-api-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:58.782367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.782361    4278 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e873cfdf-044e-41a6-86dc-14f8c4724748-metrics-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:58.782367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.782370    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7cm7\" (UniqueName: \"kubernetes.io/projected/e873cfdf-044e-41a6-86dc-14f8c4724748-kube-api-access-w7cm7\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:25:58.959549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.955509    4278 generic.go:356] "Generic (PLEG): container finished" podID="e873cfdf-044e-41a6-86dc-14f8c4724748" containerID="973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff" exitCode=0
Oct 30 03:25:58.959549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.955826    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6" event={"ID":"e873cfdf-044e-41a6-86dc-14f8c4724748","Type":"ContainerDied","Data":"973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff"}
Oct 30 03:25:58.959549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.955884    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6" event={"ID":"e873cfdf-044e-41a6-86dc-14f8c4724748","Type":"ContainerDied","Data":"ecd33a1f398a2cbca32b6812c5c8c09f6a503b335988c4de010fe5a2480d3e37"}
Oct 30 03:25:58.959549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.955914    4278 scope.go:122] "RemoveContainer" containerID="973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff"
Oct 30 03:25:58.959549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:58.956073    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"
Oct 30 03:25:59.010731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:59.009619    4278 scope.go:122] "RemoveContainer" containerID="973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff"
Oct 30 03:25:59.010731 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:25:59.010007    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff\": container with ID starting with 973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff not found: ID does not exist" containerID="973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff"
Oct 30 03:25:59.010731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:59.010044    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff"} err="failed to get container status \"973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff\": rpc error: code = NotFound desc = could not find container \"973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff\": container with ID starting with 973728fa7b7a8e5eff43f2d79050417ccb84f66152da1cb6111765dcb9a1eaff not found: ID does not exist"
Oct 30 03:25:59.032205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:59.032160    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"]
Oct 30 03:25:59.041079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:59.041045    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"]
Oct 30 03:25:59.044071 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:59.044039    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-146bb7-xwsnl"]
Oct 30 03:25:59.047053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:59.047030    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-146bb7-6ddb5d5bc9-jqvv6"]
Oct 30 03:25:59.712205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:59.712169    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"]
Oct 30 03:25:59.737449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:25:59.732235    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a0f7e-zc25m"]
Oct 30 03:26:00.199711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:00.199680    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5abf941d-6159-4046-997b-3d47675f91c4" path="/var/lib/kubelet/pods/5abf941d-6159-4046-997b-3d47675f91c4/volumes"
Oct 30 03:26:00.200153 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:00.200138    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09c888d-67bc-42b4-adb4-fc8ec1a9924b" path="/var/lib/kubelet/pods/a09c888d-67bc-42b4-adb4-fc8ec1a9924b/volumes"
Oct 30 03:26:00.200610 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:00.200594    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e873cfdf-044e-41a6-86dc-14f8c4724748" path="/var/lib/kubelet/pods/e873cfdf-044e-41a6-86dc-14f8c4724748/volumes"
Oct 30 03:26:03.751231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:03.751190    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"]
Oct 30 03:26:03.751746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:03.751460    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy" containerID="cri-o://b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3" gracePeriod=30
Oct 30 03:26:04.833095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:04.833062    4278 scope.go:122] "RemoveContainer" containerID="baec05321b320b792329f2035f4009274cc8370fed381afc8b86104f17cbba78"
Oct 30 03:26:04.843003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:04.842980    4278 scope.go:122] "RemoveContainer" containerID="8e96d57b2cf66a1cca0498df9f516fc317b274ad0516e1b86407ec4cd0f04d93"
Oct 30 03:26:04.852390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:04.852369    4278 scope.go:122] "RemoveContainer" containerID="ad126baef012f2f0c2b7354e9cf5c30a697457c30d1f25b5cf7d74497debbe9d"
Oct 30 03:26:04.864013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:04.863945    4278 scope.go:122] "RemoveContainer" containerID="ac4fe84b0c99a6e2e39e40cf48671dd0c0c1812919b690b4bcfee7b4004af350"
Oct 30 03:26:04.876062 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:04.874919    4278 scope.go:122] "RemoveContainer" containerID="5fa54f090ad0b26906c94863abb2a739e113dd0e74c2a31799dd4280d22f92ee"
Oct 30 03:26:04.887885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:04.887852    4278 scope.go:122] "RemoveContainer" containerID="864cb6c8065b638e2aa30e37ff20b7d6f40a96ae0d72132f03bf9613131a71df"
Oct 30 03:26:06.523995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.523961    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"]
Oct 30 03:26:06.524590 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.524572    4278 cpu_manager.go:414] "containerMap: removing container" podUID="e873cfdf-044e-41a6-86dc-14f8c4724748" containerName="maas-api"
Oct 30 03:26:06.524669 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.524591    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e873cfdf-044e-41a6-86dc-14f8c4724748" containerName="maas-api"
Oct 30 03:26:06.524716 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.524705    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="e873cfdf-044e-41a6-86dc-14f8c4724748" containerName="maas-api"
Oct 30 03:26:06.529978 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.529958    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.546485 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.546463    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"]
Oct 30 03:26:06.587020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.586987    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:26:06.587186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.587168    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:26:06.622007 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.621978    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.622226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.622208    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.622276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.622258    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxjc\" (UniqueName: \"kubernetes.io/projected/3541151d-ee81-4b13-ade2-9dfc878ee425-kube-api-access-pzxjc\") pod \"maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.723804 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.723768    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.723986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.723819    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxjc\" (UniqueName: \"kubernetes.io/projected/3541151d-ee81-4b13-ade2-9dfc878ee425-kube-api-access-pzxjc\") pod \"maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.723986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.723849    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.724556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.724534    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.726258 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.726237    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.748654 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.748622    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxjc\" (UniqueName: \"kubernetes.io/projected/3541151d-ee81-4b13-ade2-9dfc878ee425-kube-api-access-pzxjc\") pod \"maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.844556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.844475    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:06.897690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.897652    4278 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:26:06.897880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:06.897733    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:26:07.158252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:07.158220    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"]
Oct 30 03:26:07.159534 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:26:07.159503    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3541151d_ee81_4b13_ade2_9dfc878ee425.slice/crio-8eb1cc084961e44065343e8ca95ffd4b722fa343f621afb85b0d6113c10fa5cb WatchSource:0}: Error finding container 8eb1cc084961e44065343e8ca95ffd4b722fa343f621afb85b0d6113c10fa5cb: Status 404 returned error can't find the container with id 8eb1cc084961e44065343e8ca95ffd4b722fa343f621afb85b0d6113c10fa5cb
Oct 30 03:26:07.977472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:07.977376    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"]
Oct 30 03:26:07.981512 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:07.981493    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.004208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.004179    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"]
Oct 30 03:26:08.051923 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.051890    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.052117 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.052094    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.052169 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.052136    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6pj\" (UniqueName: \"kubernetes.io/projected/15296eb4-b666-4c66-bd0f-e21b55d0d51e-kube-api-access-mr6pj\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.101688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.101645    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9" event={"ID":"3541151d-ee81-4b13-ade2-9dfc878ee425","Type":"ContainerStarted","Data":"64f46d2d3361f9b1e689a3f0431caf9004df7f342d9255f73aa85be8d3558420"}
Oct 30 03:26:08.101854 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.101715    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9" event={"ID":"3541151d-ee81-4b13-ade2-9dfc878ee425","Type":"ContainerStarted","Data":"8eb1cc084961e44065343e8ca95ffd4b722fa343f621afb85b0d6113c10fa5cb"}
Oct 30 03:26:08.145952 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.145897    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9" podStartSLOduration=2.145883653 podStartE2EDuration="2.145883653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:08.142339005 +0000 UTC m=+2706.066547194" watchObservedRunningTime="2026-10-30 03:26:08.145883653 +0000 UTC m=+2706.070091840"
Oct 30 03:26:08.153173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.153140    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.153331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.153184    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6pj\" (UniqueName: \"kubernetes.io/projected/15296eb4-b666-4c66-bd0f-e21b55d0d51e-kube-api-access-mr6pj\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.153331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.153256    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.153947 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.153923    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.155640 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.155618    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.173090 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.173062    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6pj\" (UniqueName: \"kubernetes.io/projected/15296eb4-b666-4c66-bd0f-e21b55d0d51e-kube-api-access-mr6pj\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.296632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.296605    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:08.602839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:08.602808    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"]
Oct 30 03:26:08.609568 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:26:08.609542    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15296eb4_b666_4c66_bd0f_e21b55d0d51e.slice/crio-0a9d3f5be4b8285a546b93bbf7a216e4a2cc65310e67cfe4837b3362eb956b91 WatchSource:0}: Error finding container 0a9d3f5be4b8285a546b93bbf7a216e4a2cc65310e67cfe4837b3362eb956b91: Status 404 returned error can't find the container with id 0a9d3f5be4b8285a546b93bbf7a216e4a2cc65310e67cfe4837b3362eb956b91
Oct 30 03:26:09.032208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.032184    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 03:26:09.066444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.066408    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-kube-api-access-vr492\" (UniqueName: \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-kube-api-access-vr492\") pod \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") "
Oct 30 03:26:09.066564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.066474    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-data\") pod \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") "
Oct 30 03:26:09.066564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.066518    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-podinfo\") pod \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") "
Oct 30 03:26:09.066687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.066569    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-socket\") pod \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") "
Oct 30 03:26:09.066812 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.066783    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-token\" (UniqueName: \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-token\") pod \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") "
Oct 30 03:26:09.066935 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.066833    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-credential-socket\") pod \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") "
Oct 30 03:26:09.066935 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.066875    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-certs\") pod \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") "
Oct 30 03:26:09.067149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.066959    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istiod-ca-cert\") pod \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") "
Oct 30 03:26:09.067149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.066964    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-socket" pod "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" (UID: "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:09.067149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.067049    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-envoy\") pod \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\" (UID: \"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f\") "
Oct 30 03:26:09.067149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.067051    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-data" pod "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" (UID: "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:09.067383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.067195    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-credential-socket" pod "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" (UID: "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:09.067383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.067276    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-certs" pod "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" (UID: "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:09.067593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.067563    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istiod-ca-cert" pod "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" (UID: "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:09.067746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.067726    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.067837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.067750    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.067837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.067766    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.067837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.067779    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.067837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.067792    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.068717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.068693    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-kube-api-access-vr492" pod "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" (UID: "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f"). InnerVolumeSpecName "kube-api-access-vr492". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:09.069130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.069107    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-podinfo" pod "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" (UID: "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:26:09.069458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.069398    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-envoy" pod "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" (UID: "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:09.069919 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.069892    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-token" pod "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" (UID: "c71629bd-8e5e-4a49-a565-2cb10bc1aa1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:09.189508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.185753    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.189508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.185782    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.189508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.185794    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr492\" (UniqueName: \"kubernetes.io/projected/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-kube-api-access-vr492\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.189508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.185804    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.200494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.190118    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947" event={"ID":"15296eb4-b666-4c66-bd0f-e21b55d0d51e","Type":"ContainerStarted","Data":"49282fd5c3776795498c3f92ee13e26342bfd59777e32ff7ab3c447e6cf0ab14"}
Oct 30 03:26:09.200494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.190183    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947" event={"ID":"15296eb4-b666-4c66-bd0f-e21b55d0d51e","Type":"ContainerStarted","Data":"0a9d3f5be4b8285a546b93bbf7a216e4a2cc65310e67cfe4837b3362eb956b91"}
Oct 30 03:26:09.200494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.191822    4278 generic.go:356] "Generic (PLEG): container finished" podID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerID="b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3" exitCode=0
Oct 30 03:26:09.200494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.191864    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"
Oct 30 03:26:09.200494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.191967    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" event={"ID":"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f","Type":"ContainerDied","Data":"b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3"}
Oct 30 03:26:09.200494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.192002    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw" event={"ID":"c71629bd-8e5e-4a49-a565-2cb10bc1aa1f","Type":"ContainerDied","Data":"0ed2508b9c518aa5dfeb26655fdaeb54f964f6c90e81408a3621d836fd8777eb"}
Oct 30 03:26:09.200494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.192027    4278 scope.go:122] "RemoveContainer" containerID="b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3"
Oct 30 03:26:09.200494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.195098    4278 generic.go:356] "Generic (PLEG): container finished" podID="3541151d-ee81-4b13-ade2-9dfc878ee425" containerID="64f46d2d3361f9b1e689a3f0431caf9004df7f342d9255f73aa85be8d3558420" exitCode=0
Oct 30 03:26:09.200494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.195142    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9" event={"ID":"3541151d-ee81-4b13-ade2-9dfc878ee425","Type":"ContainerDied","Data":"64f46d2d3361f9b1e689a3f0431caf9004df7f342d9255f73aa85be8d3558420"}
Oct 30 03:26:09.213717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.210589    4278 scope.go:122] "RemoveContainer" containerID="b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3"
Oct 30 03:26:09.213717 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:26:09.210978    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3\": container with ID starting with b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3 not found: ID does not exist" containerID="b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3"
Oct 30 03:26:09.213717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.211013    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3"} err="failed to get container status \"b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3\": rpc error: code = NotFound desc = could not find container \"b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3\": container with ID starting with b75e1eb580f85b7339725fca1829442423560ce576cab078f5a2171ca76314d3 not found: ID does not exist"
Oct 30 03:26:09.237301 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.237242    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947" podStartSLOduration=2.237226605 podStartE2EDuration="2.237226605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:09.234376147 +0000 UTC m=+2707.158584676" watchObservedRunningTime="2026-10-30 03:26:09.237226605 +0000 UTC m=+2707.161434793"
Oct 30 03:26:09.311655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.311618    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"]
Oct 30 03:26:09.319135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:09.319105    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-q5mfw"]
Oct 30 03:26:10.199652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.199619    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" path="/var/lib/kubelet/pods/c71629bd-8e5e-4a49-a565-2cb10bc1aa1f/volumes"
Oct 30 03:26:10.205204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.205172    4278 generic.go:356] "Generic (PLEG): container finished" podID="15296eb4-b666-4c66-bd0f-e21b55d0d51e" containerID="49282fd5c3776795498c3f92ee13e26342bfd59777e32ff7ab3c447e6cf0ab14" exitCode=0
Oct 30 03:26:10.205355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.205249    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947" event={"ID":"15296eb4-b666-4c66-bd0f-e21b55d0d51e","Type":"ContainerDied","Data":"49282fd5c3776795498c3f92ee13e26342bfd59777e32ff7ab3c447e6cf0ab14"}
Oct 30 03:26:10.377934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.377910    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:10.502321 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.502240    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-xks-ca\") pod \"3541151d-ee81-4b13-ade2-9dfc878ee425\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") "
Oct 30 03:26:10.502498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.502385    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-openshift-ca\") pod \"3541151d-ee81-4b13-ade2-9dfc878ee425\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") "
Oct 30 03:26:10.502498 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.502477    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3541151d-ee81-4b13-ade2-9dfc878ee425-kube-api-access-pzxjc\" (UniqueName: \"kubernetes.io/projected/3541151d-ee81-4b13-ade2-9dfc878ee425-kube-api-access-pzxjc\") pod \"3541151d-ee81-4b13-ade2-9dfc878ee425\" (UID: \"3541151d-ee81-4b13-ade2-9dfc878ee425\") "
Oct 30 03:26:10.502730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.502704    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-openshift-ca" pod "3541151d-ee81-4b13-ade2-9dfc878ee425" (UID: "3541151d-ee81-4b13-ade2-9dfc878ee425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:10.503002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.502987    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:10.504327 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.504302    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-xks-ca" pod "3541151d-ee81-4b13-ade2-9dfc878ee425" (UID: "3541151d-ee81-4b13-ade2-9dfc878ee425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:10.504461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.504396    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3541151d-ee81-4b13-ade2-9dfc878ee425-kube-api-access-pzxjc" pod "3541151d-ee81-4b13-ade2-9dfc878ee425" (UID: "3541151d-ee81-4b13-ade2-9dfc878ee425"). InnerVolumeSpecName "kube-api-access-pzxjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:10.605104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.605068    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3541151d-ee81-4b13-ade2-9dfc878ee425-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:10.605104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:10.605107    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzxjc\" (UniqueName: \"kubernetes.io/projected/3541151d-ee81-4b13-ade2-9dfc878ee425-kube-api-access-pzxjc\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.215454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.215399    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"
Oct 30 03:26:11.215454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.215447    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9" event={"ID":"3541151d-ee81-4b13-ade2-9dfc878ee425","Type":"ContainerDied","Data":"8eb1cc084961e44065343e8ca95ffd4b722fa343f621afb85b0d6113c10fa5cb"}
Oct 30 03:26:11.215914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.215500    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eb1cc084961e44065343e8ca95ffd4b722fa343f621afb85b0d6113c10fa5cb"
Oct 30 03:26:11.408079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.408045    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:11.527654 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.527627    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15296eb4-b666-4c66-bd0f-e21b55d0d51e-kube-api-access-mr6pj\" (UniqueName: \"kubernetes.io/projected/15296eb4-b666-4c66-bd0f-e21b55d0d51e-kube-api-access-mr6pj\") pod \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") "
Oct 30 03:26:11.527903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.527721    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-openshift-ca\") pod \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") "
Oct 30 03:26:11.527903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.527765    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-xks-ca\") pod \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\" (UID: \"15296eb4-b666-4c66-bd0f-e21b55d0d51e\") "
Oct 30 03:26:11.528076 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.528053    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-openshift-ca" pod "15296eb4-b666-4c66-bd0f-e21b55d0d51e" (UID: "15296eb4-b666-4c66-bd0f-e21b55d0d51e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:11.528319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.528299    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.529818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.529791    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-xks-ca" pod "15296eb4-b666-4c66-bd0f-e21b55d0d51e" (UID: "15296eb4-b666-4c66-bd0f-e21b55d0d51e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:11.529923 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.529835    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15296eb4-b666-4c66-bd0f-e21b55d0d51e-kube-api-access-mr6pj" pod "15296eb4-b666-4c66-bd0f-e21b55d0d51e" (UID: "15296eb4-b666-4c66-bd0f-e21b55d0d51e"). InnerVolumeSpecName "kube-api-access-mr6pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:11.629959 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.629922    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr6pj\" (UniqueName: \"kubernetes.io/projected/15296eb4-b666-4c66-bd0f-e21b55d0d51e-kube-api-access-mr6pj\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.630126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:11.629964    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15296eb4-b666-4c66-bd0f-e21b55d0d51e-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:12.223470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:12.223417    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"
Oct 30 03:26:12.223470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:12.223450    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947" event={"ID":"15296eb4-b666-4c66-bd0f-e21b55d0d51e","Type":"ContainerDied","Data":"0a9d3f5be4b8285a546b93bbf7a216e4a2cc65310e67cfe4837b3362eb956b91"}
Oct 30 03:26:12.223983 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:12.223508    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a9d3f5be4b8285a546b93bbf7a216e4a2cc65310e67cfe4837b3362eb956b91"
Oct 30 03:26:17.526674 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.526639    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"]
Oct 30 03:26:17.527396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.527378    4278 cpu_manager.go:414] "containerMap: removing container" podUID="15296eb4-b666-4c66-bd0f-e21b55d0d51e" containerName="revoke-keys"
Oct 30 03:26:17.527499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.527399    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15296eb4-b666-4c66-bd0f-e21b55d0d51e" containerName="revoke-keys"
Oct 30 03:26:17.527499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.527446    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy"
Oct 30 03:26:17.527499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.527455    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy"
Oct 30 03:26:17.527499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.527495    4278 cpu_manager.go:414] "containerMap: removing container" podUID="3541151d-ee81-4b13-ade2-9dfc878ee425" containerName="revoke-keys"
Oct 30 03:26:17.527655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.527504    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3541151d-ee81-4b13-ade2-9dfc878ee425" containerName="revoke-keys"
Oct 30 03:26:17.527655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.527642    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="15296eb4-b666-4c66-bd0f-e21b55d0d51e" containerName="revoke-keys"
Oct 30 03:26:17.527736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.527659    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71629bd-8e5e-4a49-a565-2cb10bc1aa1f" containerName="istio-proxy"
Oct 30 03:26:17.527736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.527673    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="3541151d-ee81-4b13-ade2-9dfc878ee425" containerName="revoke-keys"
Oct 30 03:26:17.531485 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.531465    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:17.547881 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.547859    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:26:17.548002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.547926    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:26:17.553551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.553522    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"]
Oct 30 03:26:17.615357 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.615314    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbnjn\" (UniqueName: \"kubernetes.io/projected/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-kube-api-access-jbnjn\") pod \"maas-api-revoke-sub-56068983-wwr2k\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:17.615563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.615536    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56068983-wwr2k\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:17.615632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.615607    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56068983-wwr2k\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:17.717272 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.717229    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56068983-wwr2k\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:17.717499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.717370    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbnjn\" (UniqueName: \"kubernetes.io/projected/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-kube-api-access-jbnjn\") pod \"maas-api-revoke-sub-56068983-wwr2k\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:17.717594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.717512    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56068983-wwr2k\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:17.718113 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.718092    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56068983-wwr2k\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:17.719631 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.719606    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56068983-wwr2k\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:17.738359 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.738329    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbnjn\" (UniqueName: \"kubernetes.io/projected/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-kube-api-access-jbnjn\") pod \"maas-api-revoke-sub-56068983-wwr2k\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:17.845864 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:17.845781    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:18.150701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:18.150672    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"]
Oct 30 03:26:18.152073 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:26:18.152042    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2650cf5e_8ef5_4866_9cfb_af3b1f4bd0d2.slice/crio-14df39e9b1058c86a0b9bca31622394f8896409b80ca8ba0f10df2e6bf6912bf WatchSource:0}: Error finding container 14df39e9b1058c86a0b9bca31622394f8896409b80ca8ba0f10df2e6bf6912bf: Status 404 returned error can't find the container with id 14df39e9b1058c86a0b9bca31622394f8896409b80ca8ba0f10df2e6bf6912bf
Oct 30 03:26:18.277755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:18.277720    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k" event={"ID":"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2","Type":"ContainerStarted","Data":"14df39e9b1058c86a0b9bca31622394f8896409b80ca8ba0f10df2e6bf6912bf"}
Oct 30 03:26:19.285577 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:19.285543    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k" event={"ID":"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2","Type":"ContainerStarted","Data":"f574d2461c5e9d79522c9ffe9592cfd510677b7ca00dad30bc85ddbed5f6b2e4"}
Oct 30 03:26:19.326769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:19.326723    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k" podStartSLOduration=2.326709121 podStartE2EDuration="2.326709121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:19.325405294 +0000 UTC m=+2717.249613482" watchObservedRunningTime="2026-10-30 03:26:19.326709121 +0000 UTC m=+2717.250917308"
Oct 30 03:26:20.293594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:20.293558    4278 generic.go:356] "Generic (PLEG): container finished" podID="2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2" containerID="f574d2461c5e9d79522c9ffe9592cfd510677b7ca00dad30bc85ddbed5f6b2e4" exitCode=0
Oct 30 03:26:20.294039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:20.293627    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k" event={"ID":"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2","Type":"ContainerDied","Data":"f574d2461c5e9d79522c9ffe9592cfd510677b7ca00dad30bc85ddbed5f6b2e4"}
Oct 30 03:26:21.468070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:21.466894    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:21.577469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:21.577377    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-xks-ca\") pod \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") "
Oct 30 03:26:21.577692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:21.577505    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-kube-api-access-jbnjn\" (UniqueName: \"kubernetes.io/projected/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-kube-api-access-jbnjn\") pod \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") "
Oct 30 03:26:21.577692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:21.577551    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-openshift-ca\") pod \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\" (UID: \"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2\") "
Oct 30 03:26:21.577967 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:21.577941    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-openshift-ca" pod "2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2" (UID: "2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:21.578057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:21.578033    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:21.579679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:21.579643    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-xks-ca" pod "2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2" (UID: "2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:21.580002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:21.579971    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-kube-api-access-jbnjn" pod "2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2" (UID: "2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2"). InnerVolumeSpecName "kube-api-access-jbnjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:21.679022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:21.678987    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbnjn\" (UniqueName: \"kubernetes.io/projected/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-kube-api-access-jbnjn\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:21.679022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:21.679017    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:22.308282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:22.308246    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k" event={"ID":"2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2","Type":"ContainerDied","Data":"14df39e9b1058c86a0b9bca31622394f8896409b80ca8ba0f10df2e6bf6912bf"}
Oct 30 03:26:22.308457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:22.308299    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14df39e9b1058c86a0b9bca31622394f8896409b80ca8ba0f10df2e6bf6912bf"
Oct 30 03:26:22.308457 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:22.308263    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"
Oct 30 03:26:24.189810 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:26:24.189776    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:26:27.805886 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.805847    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"]
Oct 30 03:26:27.806536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.806516    4278 cpu_manager.go:414] "containerMap: removing container" podUID="2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2" containerName="revoke-keys"
Oct 30 03:26:27.806536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.806537    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2" containerName="revoke-keys"
Oct 30 03:26:27.806664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.806644    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2" containerName="revoke-keys"
Oct 30 03:26:27.810512 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.810492    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:27.839502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.839460    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:26:27.839712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.839672    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:26:27.845302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.845268    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"]
Oct 30 03:26:27.970758 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.970719    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnvk6\" (UniqueName: \"kubernetes.io/projected/00c847b6-9cfe-449f-bf15-184a7b1fa761-kube-api-access-vnvk6\") pod \"maas-api-revoke-sub-25262a2e-tqg2v\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:27.970921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.970838    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25262a2e-tqg2v\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:27.970986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:27.970968    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25262a2e-tqg2v\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:28.072379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:28.072280    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvk6\" (UniqueName: \"kubernetes.io/projected/00c847b6-9cfe-449f-bf15-184a7b1fa761-kube-api-access-vnvk6\") pod \"maas-api-revoke-sub-25262a2e-tqg2v\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:28.072379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:28.072354    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25262a2e-tqg2v\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:28.072660 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:28.072466    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25262a2e-tqg2v\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:28.073145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:28.073113    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25262a2e-tqg2v\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:28.075229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:28.075200    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25262a2e-tqg2v\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:28.095871 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:28.095835    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvk6\" (UniqueName: \"kubernetes.io/projected/00c847b6-9cfe-449f-bf15-184a7b1fa761-kube-api-access-vnvk6\") pod \"maas-api-revoke-sub-25262a2e-tqg2v\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:28.124682 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:28.124642    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:28.431727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:28.431690    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"]
Oct 30 03:26:28.432890 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:26:28.432856    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c847b6_9cfe_449f_bf15_184a7b1fa761.slice/crio-3e958ac7fa75f381e144596d232ca620b9800049a53c4862f76fc96c4ca94677 WatchSource:0}: Error finding container 3e958ac7fa75f381e144596d232ca620b9800049a53c4862f76fc96c4ca94677: Status 404 returned error can't find the container with id 3e958ac7fa75f381e144596d232ca620b9800049a53c4862f76fc96c4ca94677
Oct 30 03:26:29.367709 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:29.367670    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v" event={"ID":"00c847b6-9cfe-449f-bf15-184a7b1fa761","Type":"ContainerStarted","Data":"bc03c1cbdc0bd87a2ef56a818a0fb3e6ac0e363366a6ce2178fda747d60a3347"}
Oct 30 03:26:29.368234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:29.367743    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v" event={"ID":"00c847b6-9cfe-449f-bf15-184a7b1fa761","Type":"ContainerStarted","Data":"3e958ac7fa75f381e144596d232ca620b9800049a53c4862f76fc96c4ca94677"}
Oct 30 03:26:29.411123 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:29.411068    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v" podStartSLOduration=2.411053837 podStartE2EDuration="2.411053837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:29.406984717 +0000 UTC m=+2727.331192906" watchObservedRunningTime="2026-10-30 03:26:29.411053837 +0000 UTC m=+2727.335262025"
Oct 30 03:26:30.376494 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:30.376445    4278 generic.go:356] "Generic (PLEG): container finished" podID="00c847b6-9cfe-449f-bf15-184a7b1fa761" containerID="bc03c1cbdc0bd87a2ef56a818a0fb3e6ac0e363366a6ce2178fda747d60a3347" exitCode=0
Oct 30 03:26:30.376921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:30.376492    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v" event={"ID":"00c847b6-9cfe-449f-bf15-184a7b1fa761","Type":"ContainerDied","Data":"bc03c1cbdc0bd87a2ef56a818a0fb3e6ac0e363366a6ce2178fda747d60a3347"}
Oct 30 03:26:31.536865 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:31.536836    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:31.611827 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:31.611783    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c847b6-9cfe-449f-bf15-184a7b1fa761-kube-api-access-vnvk6\" (UniqueName: \"kubernetes.io/projected/00c847b6-9cfe-449f-bf15-184a7b1fa761-kube-api-access-vnvk6\") pod \"00c847b6-9cfe-449f-bf15-184a7b1fa761\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") "
Oct 30 03:26:31.612020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:31.611975    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-openshift-ca\") pod \"00c847b6-9cfe-449f-bf15-184a7b1fa761\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") "
Oct 30 03:26:31.612020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:31.612009    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-xks-ca\") pod \"00c847b6-9cfe-449f-bf15-184a7b1fa761\" (UID: \"00c847b6-9cfe-449f-bf15-184a7b1fa761\") "
Oct 30 03:26:31.612637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:31.612378    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-openshift-ca" pod "00c847b6-9cfe-449f-bf15-184a7b1fa761" (UID: "00c847b6-9cfe-449f-bf15-184a7b1fa761"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:31.612637 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:31.612570    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:31.614775 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:31.614747    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-xks-ca" pod "00c847b6-9cfe-449f-bf15-184a7b1fa761" (UID: "00c847b6-9cfe-449f-bf15-184a7b1fa761"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:31.614893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:31.614773    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c847b6-9cfe-449f-bf15-184a7b1fa761-kube-api-access-vnvk6" pod "00c847b6-9cfe-449f-bf15-184a7b1fa761" (UID: "00c847b6-9cfe-449f-bf15-184a7b1fa761"). InnerVolumeSpecName "kube-api-access-vnvk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:31.713741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:31.713645    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnvk6\" (UniqueName: \"kubernetes.io/projected/00c847b6-9cfe-449f-bf15-184a7b1fa761-kube-api-access-vnvk6\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:31.713741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:31.713688    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c847b6-9cfe-449f-bf15-184a7b1fa761-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:32.392503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:32.392469    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"
Oct 30 03:26:32.392503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:32.392490    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v" event={"ID":"00c847b6-9cfe-449f-bf15-184a7b1fa761","Type":"ContainerDied","Data":"3e958ac7fa75f381e144596d232ca620b9800049a53c4862f76fc96c4ca94677"}
Oct 30 03:26:32.392765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:32.392540    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e958ac7fa75f381e144596d232ca620b9800049a53c4862f76fc96c4ca94677"
Oct 30 03:26:35.349580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:35.349543    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"]
Oct 30 03:26:35.361665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:35.361628    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-723ffc-bx947"]
Oct 30 03:26:36.200050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:36.200005    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15296eb4-b666-4c66-bd0f-e21b55d0d51e" path="/var/lib/kubelet/pods/15296eb4-b666-4c66-bd0f-e21b55d0d51e/volumes"
Oct 30 03:26:41.274678 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:41.274644    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:26:42.843997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:42.843956    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"]
Oct 30 03:26:42.844576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:42.844557    4278 cpu_manager.go:414] "containerMap: removing container" podUID="00c847b6-9cfe-449f-bf15-184a7b1fa761" containerName="revoke-keys"
Oct 30 03:26:42.844576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:42.844576    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c847b6-9cfe-449f-bf15-184a7b1fa761" containerName="revoke-keys"
Oct 30 03:26:42.844712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:42.844695    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c847b6-9cfe-449f-bf15-184a7b1fa761" containerName="revoke-keys"
Oct 30 03:26:42.849555 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:42.849524    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:42.869107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:42.869077    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"]
Oct 30 03:26:42.878804 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:42.878776    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:26:42.878804 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:42.878785    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:26:43.010055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.010012    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:43.010279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.010195    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd4qm\" (UniqueName: \"kubernetes.io/projected/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-kube-api-access-fd4qm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:43.010362 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.010335    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:43.112241 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.112141    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:43.112241 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.112239    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:43.112465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.112316    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd4qm\" (UniqueName: \"kubernetes.io/projected/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-kube-api-access-fd4qm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:43.112885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.112859    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:43.114701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.114677    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:43.146120 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.146088    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd4qm\" (UniqueName: \"kubernetes.io/projected/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-kube-api-access-fd4qm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:43.164813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.164782    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:43.500533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.500497    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"]
Oct 30 03:26:43.502121 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:26:43.502088    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdeff54_aedb_493b_b8c7_b3e1d4f738d4.slice/crio-59f6a4f73f8abb607487a9f44380ef24be7188ef16a979e261880f8b7224e10a WatchSource:0}: Error finding container 59f6a4f73f8abb607487a9f44380ef24be7188ef16a979e261880f8b7224e10a: Status 404 returned error can't find the container with id 59f6a4f73f8abb607487a9f44380ef24be7188ef16a979e261880f8b7224e10a
Oct 30 03:26:43.560559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:43.560510    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs" event={"ID":"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4","Type":"ContainerStarted","Data":"59f6a4f73f8abb607487a9f44380ef24be7188ef16a979e261880f8b7224e10a"}
Oct 30 03:26:44.569650 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:44.569606    4278 generic.go:356] "Generic (PLEG): container finished" podID="5fdeff54-aedb-493b-b8c7-b3e1d4f738d4" containerID="4fd35dcfdff1d1d45e2e6c53d9b647a2badf17a10b98083e7ce18db76087ae9e" exitCode=0
Oct 30 03:26:44.570107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:44.569669    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs" event={"ID":"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4","Type":"ContainerDied","Data":"4fd35dcfdff1d1d45e2e6c53d9b647a2badf17a10b98083e7ce18db76087ae9e"}
Oct 30 03:26:45.092921 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.092866    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"]
Oct 30 03:26:45.094973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.094554    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf" podUID="fae1d571-6e0f-48c0-b28a-e7caa6056527" containerName="payload-pre-processing" containerID="cri-o://171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1" gracePeriod=30
Oct 30 03:26:45.502352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.502304    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:26:45.568744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.568709    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae1d571-6e0f-48c0-b28a-e7caa6056527-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae1d571-6e0f-48c0-b28a-e7caa6056527-plugins-config-volume\") pod \"fae1d571-6e0f-48c0-b28a-e7caa6056527\" (UID: \"fae1d571-6e0f-48c0-b28a-e7caa6056527\") "
Oct 30 03:26:45.569130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.569104    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae1d571-6e0f-48c0-b28a-e7caa6056527-plugins-config-volume" pod "fae1d571-6e0f-48c0-b28a-e7caa6056527" (UID: "fae1d571-6e0f-48c0-b28a-e7caa6056527"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:45.581046 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.581004    4278 generic.go:356] "Generic (PLEG): container finished" podID="fae1d571-6e0f-48c0-b28a-e7caa6056527" containerID="171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1" exitCode=0
Oct 30 03:26:45.581577 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.581073    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"
Oct 30 03:26:45.581577 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.581131    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf" event={"ID":"fae1d571-6e0f-48c0-b28a-e7caa6056527","Type":"ContainerDied","Data":"171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1"}
Oct 30 03:26:45.581577 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.581185    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf" event={"ID":"fae1d571-6e0f-48c0-b28a-e7caa6056527","Type":"ContainerDied","Data":"4c8914488c3d4ef6502a78157e31d17127fc2d6838992670ab2d4d28113e5b60"}
Oct 30 03:26:45.581577 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.581219    4278 scope.go:122] "RemoveContainer" containerID="171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1"
Oct 30 03:26:45.596592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.596561    4278 scope.go:122] "RemoveContainer" containerID="171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1"
Oct 30 03:26:45.597010 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:26:45.596977    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1\": container with ID starting with 171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1 not found: ID does not exist" containerID="171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1"
Oct 30 03:26:45.597130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.597018    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1"} err="failed to get container status \"171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1\": rpc error: code = NotFound desc = could not find container \"171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1\": container with ID starting with 171fad45e2d5f38930e14d9ccf678b3396f8bf18723369614d46e18fc61ecdb1 not found: ID does not exist"
Oct 30 03:26:45.670087 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.670053    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae1d571-6e0f-48c0-b28a-e7caa6056527-kube-api-access-dbbmz\" (UniqueName: \"kubernetes.io/projected/fae1d571-6e0f-48c0-b28a-e7caa6056527-kube-api-access-dbbmz\") pod \"fae1d571-6e0f-48c0-b28a-e7caa6056527\" (UID: \"fae1d571-6e0f-48c0-b28a-e7caa6056527\") "
Oct 30 03:26:45.670582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.670561    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae1d571-6e0f-48c0-b28a-e7caa6056527-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:45.672286 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.672254    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae1d571-6e0f-48c0-b28a-e7caa6056527-kube-api-access-dbbmz" pod "fae1d571-6e0f-48c0-b28a-e7caa6056527" (UID: "fae1d571-6e0f-48c0-b28a-e7caa6056527"). InnerVolumeSpecName "kube-api-access-dbbmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:45.740510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.740324    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"]
Oct 30 03:26:45.747214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.747176    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-948965-z5wh9"]
Oct 30 03:26:45.771588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.771563    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:45.771731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.771664    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbbmz\" (UniqueName: \"kubernetes.io/projected/fae1d571-6e0f-48c0-b28a-e7caa6056527-kube-api-access-dbbmz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:45.872354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.872322    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-openshift-ca\") pod \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") "
Oct 30 03:26:45.872616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.872380    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-xks-ca\") pod \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") "
Oct 30 03:26:45.872616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.872457    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-kube-api-access-fd4qm\" (UniqueName: \"kubernetes.io/projected/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-kube-api-access-fd4qm\") pod \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\" (UID: \"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4\") "
Oct 30 03:26:45.872866 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.872841    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-openshift-ca" pod "5fdeff54-aedb-493b-b8c7-b3e1d4f738d4" (UID: "5fdeff54-aedb-493b-b8c7-b3e1d4f738d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:45.873088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.873065    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:45.875082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.875053    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-kube-api-access-fd4qm" pod "5fdeff54-aedb-493b-b8c7-b3e1d4f738d4" (UID: "5fdeff54-aedb-493b-b8c7-b3e1d4f738d4"). InnerVolumeSpecName "kube-api-access-fd4qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:45.875176 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.875086    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-xks-ca" pod "5fdeff54-aedb-493b-b8c7-b3e1d4f738d4" (UID: "5fdeff54-aedb-493b-b8c7-b3e1d4f738d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:45.957699 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.957615    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"]
Oct 30 03:26:45.980870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.975218    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948965-6686d85647-4kxzf"]
Oct 30 03:26:45.980870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.975284    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd4qm\" (UniqueName: \"kubernetes.io/projected/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-kube-api-access-fd4qm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:45.980870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:45.975324    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:46.199656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:46.199618    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3541151d-ee81-4b13-ade2-9dfc878ee425" path="/var/lib/kubelet/pods/3541151d-ee81-4b13-ade2-9dfc878ee425/volumes"
Oct 30 03:26:46.200166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:46.200150    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae1d571-6e0f-48c0-b28a-e7caa6056527" path="/var/lib/kubelet/pods/fae1d571-6e0f-48c0-b28a-e7caa6056527/volumes"
Oct 30 03:26:46.590229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:46.590185    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"
Oct 30 03:26:46.590229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:46.590220    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs" event={"ID":"5fdeff54-aedb-493b-b8c7-b3e1d4f738d4","Type":"ContainerDied","Data":"59f6a4f73f8abb607487a9f44380ef24be7188ef16a979e261880f8b7224e10a"}
Oct 30 03:26:46.590883 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:46.590282    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59f6a4f73f8abb607487a9f44380ef24be7188ef16a979e261880f8b7224e10a"
Oct 30 03:26:53.247556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.247520    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"]
Oct 30 03:26:53.248089 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.247805    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy" containerID="cri-o://d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1" gracePeriod=30
Oct 30 03:26:53.794231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.794189    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"]
Oct 30 03:26:53.794791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.794776    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5fdeff54-aedb-493b-b8c7-b3e1d4f738d4" containerName="revoke-keys"
Oct 30 03:26:53.794858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.794792    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdeff54-aedb-493b-b8c7-b3e1d4f738d4" containerName="revoke-keys"
Oct 30 03:26:53.794858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.794826    4278 cpu_manager.go:414] "containerMap: removing container" podUID="fae1d571-6e0f-48c0-b28a-e7caa6056527" containerName="payload-pre-processing"
Oct 30 03:26:53.794858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.794832    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae1d571-6e0f-48c0-b28a-e7caa6056527" containerName="payload-pre-processing"
Oct 30 03:26:53.794965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.794917    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdeff54-aedb-493b-b8c7-b3e1d4f738d4" containerName="revoke-keys"
Oct 30 03:26:53.794965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.794930    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae1d571-6e0f-48c0-b28a-e7caa6056527" containerName="payload-pre-processing"
Oct 30 03:26:53.799810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.799783    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:53.817208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.817177    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:26:53.817375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.817248    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:26:53.820281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.820246    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"]
Oct 30 03:26:53.901832 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.901787    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:53.902018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.901919    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:53.902095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:53.902081    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdlc\" (UniqueName: \"kubernetes.io/projected/117acc3b-dff7-41eb-93f2-81b2ea0285c4-kube-api-access-2gdlc\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:54.003502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:54.003455    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdlc\" (UniqueName: \"kubernetes.io/projected/117acc3b-dff7-41eb-93f2-81b2ea0285c4-kube-api-access-2gdlc\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:54.003673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:54.003564    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:54.003673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:54.003617    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:54.004236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:54.004208    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:54.005993 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:54.005969    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:54.024090 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:54.024058    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdlc\" (UniqueName: \"kubernetes.io/projected/117acc3b-dff7-41eb-93f2-81b2ea0285c4-kube-api-access-2gdlc\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:54.115243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:54.115137    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:54.205699 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:26:54.205661    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:26:54.430780 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:54.430750    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"]
Oct 30 03:26:54.432802 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:26:54.432766    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117acc3b_dff7_41eb_93f2_81b2ea0285c4.slice/crio-aa9085913d4fddf2697cc7d9424c7658fa0b29017e7340c63e6f6999d90dfb36 WatchSource:0}: Error finding container aa9085913d4fddf2697cc7d9424c7658fa0b29017e7340c63e6f6999d90dfb36: Status 404 returned error can't find the container with id aa9085913d4fddf2697cc7d9424c7658fa0b29017e7340c63e6f6999d90dfb36
Oct 30 03:26:54.654709 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:54.654664    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24" event={"ID":"117acc3b-dff7-41eb-93f2-81b2ea0285c4","Type":"ContainerStarted","Data":"aa9085913d4fddf2697cc7d9424c7658fa0b29017e7340c63e6f6999d90dfb36"}
Oct 30 03:26:55.664126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:55.664028    4278 generic.go:356] "Generic (PLEG): container finished" podID="117acc3b-dff7-41eb-93f2-81b2ea0285c4" containerID="6abb5cfe2166f4e5cab752e2a706076b73942fb4b3fa9f2136eff74d78847f1f" exitCode=0
Oct 30 03:26:55.664518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:55.664118    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24" event={"ID":"117acc3b-dff7-41eb-93f2-81b2ea0285c4","Type":"ContainerDied","Data":"6abb5cfe2166f4e5cab752e2a706076b73942fb4b3fa9f2136eff74d78847f1f"}
Oct 30 03:26:56.819735 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:56.819702    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:56.858691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:56.858657    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-openshift-ca\") pod \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") "
Oct 30 03:26:56.858928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:56.858720    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-xks-ca\") pod \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") "
Oct 30 03:26:56.858928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:56.858792    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117acc3b-dff7-41eb-93f2-81b2ea0285c4-kube-api-access-2gdlc\" (UniqueName: \"kubernetes.io/projected/117acc3b-dff7-41eb-93f2-81b2ea0285c4-kube-api-access-2gdlc\") pod \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\" (UID: \"117acc3b-dff7-41eb-93f2-81b2ea0285c4\") "
Oct 30 03:26:56.859160 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:56.859133    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-openshift-ca" pod "117acc3b-dff7-41eb-93f2-81b2ea0285c4" (UID: "117acc3b-dff7-41eb-93f2-81b2ea0285c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:56.859394 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:56.859374    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:56.861003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:56.860971    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-xks-ca" pod "117acc3b-dff7-41eb-93f2-81b2ea0285c4" (UID: "117acc3b-dff7-41eb-93f2-81b2ea0285c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:56.861114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:56.861050    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117acc3b-dff7-41eb-93f2-81b2ea0285c4-kube-api-access-2gdlc" pod "117acc3b-dff7-41eb-93f2-81b2ea0285c4" (UID: "117acc3b-dff7-41eb-93f2-81b2ea0285c4"). InnerVolumeSpecName "kube-api-access-2gdlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:56.960582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:56.960485    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117acc3b-dff7-41eb-93f2-81b2ea0285c4-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:56.960582 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:56.960537    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdlc\" (UniqueName: \"kubernetes.io/projected/117acc3b-dff7-41eb-93f2-81b2ea0285c4-kube-api-access-2gdlc\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:57.683557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:57.683509    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24" event={"ID":"117acc3b-dff7-41eb-93f2-81b2ea0285c4","Type":"ContainerDied","Data":"aa9085913d4fddf2697cc7d9424c7658fa0b29017e7340c63e6f6999d90dfb36"}
Oct 30 03:26:57.683765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:57.683589    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa9085913d4fddf2697cc7d9424c7658fa0b29017e7340c63e6f6999d90dfb36"
Oct 30 03:26:57.683765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:57.683675    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"
Oct 30 03:26:58.681870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.681836    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:26:58.694116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.694080    4278 generic.go:356] "Generic (PLEG): container finished" podID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerID="d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1" exitCode=0
Oct 30 03:26:58.694341 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.694134    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"
Oct 30 03:26:58.694341 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.694171    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" event={"ID":"db05207e-7e87-4757-a68c-e9bf54b970fc","Type":"ContainerDied","Data":"d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1"}
Oct 30 03:26:58.694341 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.694230    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r" event={"ID":"db05207e-7e87-4757-a68c-e9bf54b970fc","Type":"ContainerDied","Data":"951a6a1c99df692758fedac7a51da4874438cbcdfd6ce97ed26ca704faa5c09a"}
Oct 30 03:26:58.694341 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.694263    4278 scope.go:122] "RemoveContainer" containerID="d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1"
Oct 30 03:26:58.699182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699159    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-token\" (UniqueName: \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-token\") pod \"db05207e-7e87-4757-a68c-e9bf54b970fc\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") "
Oct 30 03:26:58.699334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699251    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-envoy\") pod \"db05207e-7e87-4757-a68c-e9bf54b970fc\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") "
Oct 30 03:26:58.699334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699326    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-podinfo\") pod \"db05207e-7e87-4757-a68c-e9bf54b970fc\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") "
Oct 30 03:26:58.699481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699371    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db05207e-7e87-4757-a68c-e9bf54b970fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db05207e-7e87-4757-a68c-e9bf54b970fc-istiod-ca-cert\") pod \"db05207e-7e87-4757-a68c-e9bf54b970fc\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") "
Oct 30 03:26:58.699481 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699440    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-socket\") pod \"db05207e-7e87-4757-a68c-e9bf54b970fc\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") "
Oct 30 03:26:58.699572 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699548    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-data\") pod \"db05207e-7e87-4757-a68c-e9bf54b970fc\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") "
Oct 30 03:26:58.699641 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699616    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-certs\") pod \"db05207e-7e87-4757-a68c-e9bf54b970fc\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") "
Oct 30 03:26:58.699774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699755    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-credential-socket\") pod \"db05207e-7e87-4757-a68c-e9bf54b970fc\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") "
Oct 30 03:26:58.699851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699827    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-kube-api-access-4jl2c\" (UniqueName: \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-kube-api-access-4jl2c\") pod \"db05207e-7e87-4757-a68c-e9bf54b970fc\" (UID: \"db05207e-7e87-4757-a68c-e9bf54b970fc\") "
Oct 30 03:26:58.699920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699865    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-data" pod "db05207e-7e87-4757-a68c-e9bf54b970fc" (UID: "db05207e-7e87-4757-a68c-e9bf54b970fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:58.699920 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699878    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db05207e-7e87-4757-a68c-e9bf54b970fc-istiod-ca-cert" pod "db05207e-7e87-4757-a68c-e9bf54b970fc" (UID: "db05207e-7e87-4757-a68c-e9bf54b970fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:58.700034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699977    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-socket" pod "db05207e-7e87-4757-a68c-e9bf54b970fc" (UID: "db05207e-7e87-4757-a68c-e9bf54b970fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:58.700034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.699996    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-certs" pod "db05207e-7e87-4757-a68c-e9bf54b970fc" (UID: "db05207e-7e87-4757-a68c-e9bf54b970fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:58.700214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.700192    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-credential-socket" pod "db05207e-7e87-4757-a68c-e9bf54b970fc" (UID: "db05207e-7e87-4757-a68c-e9bf54b970fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:58.700458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.700400    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:58.700595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.700465    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db05207e-7e87-4757-a68c-e9bf54b970fc-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:58.700595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.700481    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:58.700595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.700491    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:58.700595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.700499    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:58.701966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.701917    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-envoy" pod "db05207e-7e87-4757-a68c-e9bf54b970fc" (UID: "db05207e-7e87-4757-a68c-e9bf54b970fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:58.702123 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.702100    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-token" pod "db05207e-7e87-4757-a68c-e9bf54b970fc" (UID: "db05207e-7e87-4757-a68c-e9bf54b970fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:58.702456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.702400    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-kube-api-access-4jl2c" pod "db05207e-7e87-4757-a68c-e9bf54b970fc" (UID: "db05207e-7e87-4757-a68c-e9bf54b970fc"). InnerVolumeSpecName "kube-api-access-4jl2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:58.703713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.703691    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-podinfo" pod "db05207e-7e87-4757-a68c-e9bf54b970fc" (UID: "db05207e-7e87-4757-a68c-e9bf54b970fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:26:58.707748 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.707726    4278 scope.go:122] "RemoveContainer" containerID="83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4"
Oct 30 03:26:58.722988 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.722967    4278 scope.go:122] "RemoveContainer" containerID="d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1"
Oct 30 03:26:58.723336 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:26:58.723307    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1\": container with ID starting with d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1 not found: ID does not exist" containerID="d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1"
Oct 30 03:26:58.723404 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.723353    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1"} err="failed to get container status \"d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1\": rpc error: code = NotFound desc = could not find container \"d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1\": container with ID starting with d71981f59812458ae596ffb4e34c9cc20fd454805a12b2fdc0b6122ffe0264b1 not found: ID does not exist"
Oct 30 03:26:58.723404 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.723379    4278 scope.go:122] "RemoveContainer" containerID="83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4"
Oct 30 03:26:58.723671 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:26:58.723650    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4\": container with ID starting with 83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4 not found: ID does not exist" containerID="83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4"
Oct 30 03:26:58.723718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.723681    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4"} err="failed to get container status \"83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4\": rpc error: code = NotFound desc = could not find container \"83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4\": container with ID starting with 83dc76c65384a3d50a6c967dfdd08364892b94006ec122c9429c65325cfc08a4 not found: ID does not exist"
Oct 30 03:26:58.802166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.802130    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:58.802166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.802162    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jl2c\" (UniqueName: \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-kube-api-access-4jl2c\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:58.802166 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.802175    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:58.802456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:58.802184    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db05207e-7e87-4757-a68c-e9bf54b970fc-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:26:59.043640 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:59.043607    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"]
Oct 30 03:26:59.056299 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:26:59.056253    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-948965-openshift-default-5555b799b7-9wj5r"]
Oct 30 03:27:00.200488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:00.200451    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" path="/var/lib/kubelet/pods/db05207e-7e87-4757-a68c-e9bf54b970fc/volumes"
Oct 30 03:27:04.951723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:04.951683    4278 scope.go:122] "RemoveContainer" containerID="e65657d59e32940bf6e8c950b6029c61b9d4f6721196562101155177fcb9f317"
Oct 30 03:27:04.962269 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:04.962240    4278 scope.go:122] "RemoveContainer" containerID="26f786b0baa6297088cb62d6b1e7fd279568d78c7a342185eddda6e364f85d84"
Oct 30 03:27:04.971971 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:04.971950    4278 scope.go:122] "RemoveContainer" containerID="61c8e493e1565ef457d83ac798949783d96d55ae1b824188c3b9c7d856b4cd0a"
Oct 30 03:27:06.420388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:06.420347    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"]
Oct 30 03:27:06.432973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:06.432901    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac7936-9kg24"]
Oct 30 03:27:08.199322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:08.199284    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117acc3b-dff7-41eb-93f2-81b2ea0285c4" path="/var/lib/kubelet/pods/117acc3b-dff7-41eb-93f2-81b2ea0285c4/volumes"
Oct 30 03:27:10.980522 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:10.980483    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"]
Oct 30 03:27:10.981108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:10.980711    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn" podUID="42127ef5-f8a8-4c9d-9002-034162b8bdf4" containerName="maas-api" containerID="cri-o://ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0" gracePeriod=30
Oct 30 03:27:11.273844 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.273810    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:27:11.322327 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.322273    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-metrics-certs\") pod \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") "
Oct 30 03:27:11.322589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.322459    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-maas-api-tls\") pod \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") "
Oct 30 03:27:11.322589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.322529    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42127ef5-f8a8-4c9d-9002-034162b8bdf4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42127ef5-f8a8-4c9d-9002-034162b8bdf4-openshift-service-ca\") pod \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") "
Oct 30 03:27:11.323467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.322724    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42127ef5-f8a8-4c9d-9002-034162b8bdf4-kube-api-access-phnkw\" (UniqueName: \"kubernetes.io/projected/42127ef5-f8a8-4c9d-9002-034162b8bdf4-kube-api-access-phnkw\") pod \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\" (UID: \"42127ef5-f8a8-4c9d-9002-034162b8bdf4\") "
Oct 30 03:27:11.341966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.341683    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42127ef5-f8a8-4c9d-9002-034162b8bdf4-openshift-service-ca" pod "42127ef5-f8a8-4c9d-9002-034162b8bdf4" (UID: "42127ef5-f8a8-4c9d-9002-034162b8bdf4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:11.347005 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.345648    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-maas-api-tls" pod "42127ef5-f8a8-4c9d-9002-034162b8bdf4" (UID: "42127ef5-f8a8-4c9d-9002-034162b8bdf4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:11.361450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.360068    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-metrics-certs" pod "42127ef5-f8a8-4c9d-9002-034162b8bdf4" (UID: "42127ef5-f8a8-4c9d-9002-034162b8bdf4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:11.365450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.362602    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42127ef5-f8a8-4c9d-9002-034162b8bdf4-kube-api-access-phnkw" pod "42127ef5-f8a8-4c9d-9002-034162b8bdf4" (UID: "42127ef5-f8a8-4c9d-9002-034162b8bdf4"). InnerVolumeSpecName "kube-api-access-phnkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:11.466830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.427895    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phnkw\" (UniqueName: \"kubernetes.io/projected/42127ef5-f8a8-4c9d-9002-034162b8bdf4-kube-api-access-phnkw\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:11.466830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.427943    4278 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-metrics-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:11.466830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.427958    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42127ef5-f8a8-4c9d-9002-034162b8bdf4-maas-api-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:11.466830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.427975    4278 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42127ef5-f8a8-4c9d-9002-034162b8bdf4-openshift-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:11.695661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.695619    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"]
Oct 30 03:27:11.698645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.698616    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e2727-r7wzs"]
Oct 30 03:27:11.797889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.797846    4278 generic.go:356] "Generic (PLEG): container finished" podID="42127ef5-f8a8-4c9d-9002-034162b8bdf4" containerID="ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0" exitCode=0
Oct 30 03:27:11.798065 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.797917    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"
Oct 30 03:27:11.798065 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.797921    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn" event={"ID":"42127ef5-f8a8-4c9d-9002-034162b8bdf4","Type":"ContainerDied","Data":"ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0"}
Oct 30 03:27:11.798065 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.798013    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn" event={"ID":"42127ef5-f8a8-4c9d-9002-034162b8bdf4","Type":"ContainerDied","Data":"efbe3287206e22af15b69d7ae36f29f48d1900088bfdffce4e5b5d77a4b28320"}
Oct 30 03:27:11.798065 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.798037    4278 scope.go:122] "RemoveContainer" containerID="ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0"
Oct 30 03:27:11.809714 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.809693    4278 scope.go:122] "RemoveContainer" containerID="ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0"
Oct 30 03:27:11.810039 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:27:11.810019    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0\": container with ID starting with ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0 not found: ID does not exist" containerID="ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0"
Oct 30 03:27:11.810119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.810049    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0"} err="failed to get container status \"ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0\": rpc error: code = NotFound desc = could not find container \"ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0\": container with ID starting with ea2dcdafda214ca354747a149bb09e5b111453a66a068bc86871a00a42bdd9d0 not found: ID does not exist"
Oct 30 03:27:11.852096 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.852059    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"]
Oct 30 03:27:11.858801 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:11.858767    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e2727-6f74c549b8-zcftn"]
Oct 30 03:27:12.200138 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:12.200100    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42127ef5-f8a8-4c9d-9002-034162b8bdf4" path="/var/lib/kubelet/pods/42127ef5-f8a8-4c9d-9002-034162b8bdf4/volumes"
Oct 30 03:27:12.200616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:12.200599    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fdeff54-aedb-493b-b8c7-b3e1d4f738d4" path="/var/lib/kubelet/pods/5fdeff54-aedb-493b-b8c7-b3e1d4f738d4/volumes"
Oct 30 03:27:14.172330 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.172290    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"]
Oct 30 03:27:14.173296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.173110    4278 cpu_manager.go:414] "containerMap: removing container" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy"
Oct 30 03:27:14.173296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.173134    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy"
Oct 30 03:27:14.173296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.173154    4278 cpu_manager.go:414] "containerMap: removing container" podUID="117acc3b-dff7-41eb-93f2-81b2ea0285c4" containerName="revoke-keys"
Oct 30 03:27:14.173296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.173161    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117acc3b-dff7-41eb-93f2-81b2ea0285c4" containerName="revoke-keys"
Oct 30 03:27:14.173296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.173188    4278 cpu_manager.go:414] "containerMap: removing container" podUID="42127ef5-f8a8-4c9d-9002-034162b8bdf4" containerName="maas-api"
Oct 30 03:27:14.173296 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.173195    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42127ef5-f8a8-4c9d-9002-034162b8bdf4" containerName="maas-api"
Oct 30 03:27:14.173704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.173383    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy"
Oct 30 03:27:14.173704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.173400    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="117acc3b-dff7-41eb-93f2-81b2ea0285c4" containerName="revoke-keys"
Oct 30 03:27:14.173704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.173416    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="42127ef5-f8a8-4c9d-9002-034162b8bdf4" containerName="maas-api"
Oct 30 03:27:14.178701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.178676    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.200712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.200684    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:27:14.200712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.200701    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:27:14.204675 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.204648    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"]
Oct 30 03:27:14.268078 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.268039    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.268276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.268173    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.268276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.268222    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vlw\" (UniqueName: \"kubernetes.io/projected/1cfe0574-64e3-45bc-a097-ee984c674fa7-kube-api-access-69vlw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.369767 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.369718    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.369767 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.369782    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vlw\" (UniqueName: \"kubernetes.io/projected/1cfe0574-64e3-45bc-a097-ee984c674fa7-kube-api-access-69vlw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.370055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.369965    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.370440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.370402    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.372453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.372396    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.390662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.390628    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vlw\" (UniqueName: \"kubernetes.io/projected/1cfe0574-64e3-45bc-a097-ee984c674fa7-kube-api-access-69vlw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.492675 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.492586    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:14.801902 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.801865    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"]
Oct 30 03:27:14.806938 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:27:14.806908    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfe0574_64e3_45bc_a097_ee984c674fa7.slice/crio-246aab0c71ded8fca7a52167cf1f793e6b73cd04679e1ca1fdd31108709adc66 WatchSource:0}: Error finding container 246aab0c71ded8fca7a52167cf1f793e6b73cd04679e1ca1fdd31108709adc66: Status 404 returned error can't find the container with id 246aab0c71ded8fca7a52167cf1f793e6b73cd04679e1ca1fdd31108709adc66
Oct 30 03:27:14.832563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:14.832533    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8" event={"ID":"1cfe0574-64e3-45bc-a097-ee984c674fa7","Type":"ContainerStarted","Data":"246aab0c71ded8fca7a52167cf1f793e6b73cd04679e1ca1fdd31108709adc66"}
Oct 30 03:27:15.846344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:15.846289    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8" event={"ID":"1cfe0574-64e3-45bc-a097-ee984c674fa7","Type":"ContainerStarted","Data":"30666648d93923e5f21a2d1f7d780259ce2faeed3397a68562444c3b6f98e0ce"}
Oct 30 03:27:15.890562 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:15.890500    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8" podStartSLOduration=1.8904846100000001 podStartE2EDuration="1.89048461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:15.886830855 +0000 UTC m=+2773.811039043" watchObservedRunningTime="2026-10-30 03:27:15.89048461 +0000 UTC m=+2773.814692799"
Oct 30 03:27:16.874910 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:16.874853    4278 generic.go:356] "Generic (PLEG): container finished" podID="1cfe0574-64e3-45bc-a097-ee984c674fa7" containerID="30666648d93923e5f21a2d1f7d780259ce2faeed3397a68562444c3b6f98e0ce" exitCode=0
Oct 30 03:27:16.875635 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:16.874970    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8" event={"ID":"1cfe0574-64e3-45bc-a097-ee984c674fa7","Type":"ContainerDied","Data":"30666648d93923e5f21a2d1f7d780259ce2faeed3397a68562444c3b6f98e0ce"}
Oct 30 03:27:18.032271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.032239    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:18.126585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.126549    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-openshift-ca\") pod \"1cfe0574-64e3-45bc-a097-ee984c674fa7\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") "
Oct 30 03:27:18.126813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.126595    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-xks-ca\") pod \"1cfe0574-64e3-45bc-a097-ee984c674fa7\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") "
Oct 30 03:27:18.126813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.126629    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfe0574-64e3-45bc-a097-ee984c674fa7-kube-api-access-69vlw\" (UniqueName: \"kubernetes.io/projected/1cfe0574-64e3-45bc-a097-ee984c674fa7-kube-api-access-69vlw\") pod \"1cfe0574-64e3-45bc-a097-ee984c674fa7\" (UID: \"1cfe0574-64e3-45bc-a097-ee984c674fa7\") "
Oct 30 03:27:18.127063 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.127035    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-openshift-ca" pod "1cfe0574-64e3-45bc-a097-ee984c674fa7" (UID: "1cfe0574-64e3-45bc-a097-ee984c674fa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:18.128946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.128921    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfe0574-64e3-45bc-a097-ee984c674fa7-kube-api-access-69vlw" pod "1cfe0574-64e3-45bc-a097-ee984c674fa7" (UID: "1cfe0574-64e3-45bc-a097-ee984c674fa7"). InnerVolumeSpecName "kube-api-access-69vlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:18.129074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.128970    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-xks-ca" pod "1cfe0574-64e3-45bc-a097-ee984c674fa7" (UID: "1cfe0574-64e3-45bc-a097-ee984c674fa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:18.252768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.236654    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.252768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.236697    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cfe0574-64e3-45bc-a097-ee984c674fa7-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.252768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.236718    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69vlw\" (UniqueName: \"kubernetes.io/projected/1cfe0574-64e3-45bc-a097-ee984c674fa7-kube-api-access-69vlw\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.889323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.889295    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"
Oct 30 03:27:18.889503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.889292    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8" event={"ID":"1cfe0574-64e3-45bc-a097-ee984c674fa7","Type":"ContainerDied","Data":"246aab0c71ded8fca7a52167cf1f793e6b73cd04679e1ca1fdd31108709adc66"}
Oct 30 03:27:18.889503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:18.889465    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="246aab0c71ded8fca7a52167cf1f793e6b73cd04679e1ca1fdd31108709adc66"
Oct 30 03:27:19.095402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.095359    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"]
Oct 30 03:27:19.095951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.095933    4278 cpu_manager.go:414] "containerMap: removing container" podUID="1cfe0574-64e3-45bc-a097-ee984c674fa7" containerName="revoke-keys"
Oct 30 03:27:19.095996 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.095952    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfe0574-64e3-45bc-a097-ee984c674fa7" containerName="revoke-keys"
Oct 30 03:27:19.095996 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.095993    4278 cpu_manager.go:414] "containerMap: removing container" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy"
Oct 30 03:27:19.096061 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.096001    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy"
Oct 30 03:27:19.096111 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.096100    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfe0574-64e3-45bc-a097-ee984c674fa7" containerName="revoke-keys"
Oct 30 03:27:19.096161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.096120    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="db05207e-7e87-4757-a68c-e9bf54b970fc" containerName="istio-proxy"
Oct 30 03:27:19.099636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.099613    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.117084 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.117053    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:27:19.117270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.117246    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:27:19.121275 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.121248    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"]
Oct 30 03:27:19.252621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.252513    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzldd\" (UniqueName: \"kubernetes.io/projected/02b796c3-f76b-4baf-8d8c-1431ab0b798e-kube-api-access-dzldd\") pod \"maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.252788 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.252681    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.252849 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.252812    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.354041 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.354003    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzldd\" (UniqueName: \"kubernetes.io/projected/02b796c3-f76b-4baf-8d8c-1431ab0b798e-kube-api-access-dzldd\") pod \"maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.354260 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.354097    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.354260 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.354210    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.354954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.354924    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.356611 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.356590    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.377229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.377194    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzldd\" (UniqueName: \"kubernetes.io/projected/02b796c3-f76b-4baf-8d8c-1431ab0b798e-kube-api-access-dzldd\") pod \"maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.413786 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.413752    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:19.717281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.717254    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"]
Oct 30 03:27:19.718570 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:27:19.718544    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b796c3_f76b_4baf_8d8c_1431ab0b798e.slice/crio-a8f13125093fe6028a77e13a0340ba52985d13fbeecd31f2b4d71bf87cdaae45 WatchSource:0}: Error finding container a8f13125093fe6028a77e13a0340ba52985d13fbeecd31f2b4d71bf87cdaae45: Status 404 returned error can't find the container with id a8f13125093fe6028a77e13a0340ba52985d13fbeecd31f2b4d71bf87cdaae45
Oct 30 03:27:19.897090 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:19.897053    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc" event={"ID":"02b796c3-f76b-4baf-8d8c-1431ab0b798e","Type":"ContainerStarted","Data":"a8f13125093fe6028a77e13a0340ba52985d13fbeecd31f2b4d71bf87cdaae45"}
Oct 30 03:27:20.907466 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:20.907411    4278 generic.go:356] "Generic (PLEG): container finished" podID="02b796c3-f76b-4baf-8d8c-1431ab0b798e" containerID="19cd623ce9206333b87a9036c56cefc1c95e9ce102406d347c2b4f69a3fceafd" exitCode=0
Oct 30 03:27:20.907902 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:20.907507    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc" event={"ID":"02b796c3-f76b-4baf-8d8c-1431ab0b798e","Type":"ContainerDied","Data":"19cd623ce9206333b87a9036c56cefc1c95e9ce102406d347c2b4f69a3fceafd"}
Oct 30 03:27:22.071412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.071382    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:22.190366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.190266    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b796c3-f76b-4baf-8d8c-1431ab0b798e-kube-api-access-dzldd\" (UniqueName: \"kubernetes.io/projected/02b796c3-f76b-4baf-8d8c-1431ab0b798e-kube-api-access-dzldd\") pod \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") "
Oct 30 03:27:22.190366 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.190328    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-xks-ca\") pod \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") "
Oct 30 03:27:22.190712 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.190376    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-openshift-ca\") pod \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\" (UID: \"02b796c3-f76b-4baf-8d8c-1431ab0b798e\") "
Oct 30 03:27:22.190790 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.190770    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-openshift-ca" pod "02b796c3-f76b-4baf-8d8c-1431ab0b798e" (UID: "02b796c3-f76b-4baf-8d8c-1431ab0b798e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:22.192661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.192633    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-xks-ca" pod "02b796c3-f76b-4baf-8d8c-1431ab0b798e" (UID: "02b796c3-f76b-4baf-8d8c-1431ab0b798e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:22.192661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.192647    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b796c3-f76b-4baf-8d8c-1431ab0b798e-kube-api-access-dzldd" pod "02b796c3-f76b-4baf-8d8c-1431ab0b798e" (UID: "02b796c3-f76b-4baf-8d8c-1431ab0b798e"). InnerVolumeSpecName "kube-api-access-dzldd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:22.292283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.292246    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzldd\" (UniqueName: \"kubernetes.io/projected/02b796c3-f76b-4baf-8d8c-1431ab0b798e-kube-api-access-dzldd\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:22.292283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.292276    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:22.292283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.292286    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b796c3-f76b-4baf-8d8c-1431ab0b798e-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:22.925337 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.925301    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"
Oct 30 03:27:22.925337 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.925315    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc" event={"ID":"02b796c3-f76b-4baf-8d8c-1431ab0b798e","Type":"ContainerDied","Data":"a8f13125093fe6028a77e13a0340ba52985d13fbeecd31f2b4d71bf87cdaae45"}
Oct 30 03:27:22.925595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:22.925372    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f13125093fe6028a77e13a0340ba52985d13fbeecd31f2b4d71bf87cdaae45"
Oct 30 03:27:23.903663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:23.903622    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"]
Oct 30 03:27:23.929137 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:23.929105    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"]
Oct 30 03:27:24.031548 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:24.031508    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"]
Oct 30 03:27:24.031772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:24.031755    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5587d7ab-kp9ql"]
Oct 30 03:27:24.196907 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:27:24.196805    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:27:24.202211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:24.202182    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43c028a-5967-422c-8a5b-e740c3d279cd" path="/var/lib/kubelet/pods/f43c028a-5967-422c-8a5b-e740c3d279cd/volumes"
Oct 30 03:27:24.281720 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:24.281683    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:27:30.421832 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.421795    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"]
Oct 30 03:27:30.422325 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.422312    4278 cpu_manager.go:414] "containerMap: removing container" podUID="02b796c3-f76b-4baf-8d8c-1431ab0b798e" containerName="revoke-keys"
Oct 30 03:27:30.422369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.422326    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b796c3-f76b-4baf-8d8c-1431ab0b798e" containerName="revoke-keys"
Oct 30 03:27:30.422456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.422446    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b796c3-f76b-4baf-8d8c-1431ab0b798e" containerName="revoke-keys"
Oct 30 03:27:30.427672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.427644    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.444677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.444644    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"]
Oct 30 03:27:30.449093 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.449067    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"]
Oct 30 03:27:30.449266 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.449183    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.451300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.451271    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:27:30.456877 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.455684    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:27:30.460095 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.460068    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"]
Oct 30 03:27:30.466186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.465301    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:30.469077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.469053    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"]
Oct 30 03:27:30.487074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.487041    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"]
Oct 30 03:27:30.520345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.520303    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26b286c0-xh7j6\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.520547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.520378    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxg85\" (UniqueName: \"kubernetes.io/projected/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-kube-api-access-dxg85\") pod \"maas-api-revoke-sub-3f25a50e-7v4cz\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:30.520547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.520506    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ec6b54-7gh6q\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.520634 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.520548    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26b286c0-xh7j6\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.520710 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.520688    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzhlf\" (UniqueName: \"kubernetes.io/projected/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-kube-api-access-rzhlf\") pod \"maas-api-revoke-sub-26b286c0-xh7j6\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.520750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.520729    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f25a50e-7v4cz\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:30.520809 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.520793    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ec6b54-7gh6q\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.520864 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.520851    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nznm\" (UniqueName: \"kubernetes.io/projected/484aa77b-b879-4fe5-acad-4f1e1d0787d0-kube-api-access-7nznm\") pod \"maas-api-revoke-sub-c6ec6b54-7gh6q\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.520951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.520935    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f25a50e-7v4cz\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:30.621653 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.621613    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxg85\" (UniqueName: \"kubernetes.io/projected/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-kube-api-access-dxg85\") pod \"maas-api-revoke-sub-3f25a50e-7v4cz\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:30.621653 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.621658    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ec6b54-7gh6q\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.621914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.621763    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26b286c0-xh7j6\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.621914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.621834    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhlf\" (UniqueName: \"kubernetes.io/projected/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-kube-api-access-rzhlf\") pod \"maas-api-revoke-sub-26b286c0-xh7j6\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.621914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.621867    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f25a50e-7v4cz\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:30.621914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.621909    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ec6b54-7gh6q\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.622131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.621968    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nznm\" (UniqueName: \"kubernetes.io/projected/484aa77b-b879-4fe5-acad-4f1e1d0787d0-kube-api-access-7nznm\") pod \"maas-api-revoke-sub-c6ec6b54-7gh6q\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.622131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.622015    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f25a50e-7v4cz\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:30.622131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.622080    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26b286c0-xh7j6\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.622749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.622718    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ec6b54-7gh6q\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.622874 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.622819    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f25a50e-7v4cz\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:30.622929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.622871    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26b286c0-xh7j6\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.624535 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.624511    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26b286c0-xh7j6\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.624867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.624845    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ec6b54-7gh6q\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.624957 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.624855    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f25a50e-7v4cz\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:30.641196 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.641169    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxg85\" (UniqueName: \"kubernetes.io/projected/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-kube-api-access-dxg85\") pod \"maas-api-revoke-sub-3f25a50e-7v4cz\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:30.641319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.641174    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nznm\" (UniqueName: \"kubernetes.io/projected/484aa77b-b879-4fe5-acad-4f1e1d0787d0-kube-api-access-7nznm\") pod \"maas-api-revoke-sub-c6ec6b54-7gh6q\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.641319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.641283    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhlf\" (UniqueName: \"kubernetes.io/projected/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-kube-api-access-rzhlf\") pod \"maas-api-revoke-sub-26b286c0-xh7j6\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.745059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.744975    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:30.768668 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.768633    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:30.786231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:30.786199    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:31.061813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:31.059171    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"]
Oct 30 03:27:31.064552 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:27:31.064502    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484aa77b_b879_4fe5_acad_4f1e1d0787d0.slice/crio-a5e3d8dd34c2f4f125024cbab6b592b9371777d73d5c0680875590f98b9dbc15 WatchSource:0}: Error finding container a5e3d8dd34c2f4f125024cbab6b592b9371777d73d5c0680875590f98b9dbc15: Status 404 returned error can't find the container with id a5e3d8dd34c2f4f125024cbab6b592b9371777d73d5c0680875590f98b9dbc15
Oct 30 03:27:31.101839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:31.101801    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"]
Oct 30 03:27:31.104126 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:27:31.104096    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f43f0a6_77de_47a5_9bdb_4b4763f6eb14.slice/crio-0e0439d87f2fedd6d370b2c6905e92c9ac78fc5c7756fb8cbf412b648d3083fe WatchSource:0}: Error finding container 0e0439d87f2fedd6d370b2c6905e92c9ac78fc5c7756fb8cbf412b648d3083fe: Status 404 returned error can't find the container with id 0e0439d87f2fedd6d370b2c6905e92c9ac78fc5c7756fb8cbf412b648d3083fe
Oct 30 03:27:31.123520 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:31.123482    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"]
Oct 30 03:27:31.126388 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:27:31.126361    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1086f8f_0dde_493c_8fd2_c8fe6e2ca516.slice/crio-5789aaea6fb461da992a8ef08f65c87694f733f74804bb3a7b4d61144db2bd45 WatchSource:0}: Error finding container 5789aaea6fb461da992a8ef08f65c87694f733f74804bb3a7b4d61144db2bd45: Status 404 returned error can't find the container with id 5789aaea6fb461da992a8ef08f65c87694f733f74804bb3a7b4d61144db2bd45
Oct 30 03:27:31.991181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:31.991143    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6" event={"ID":"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14","Type":"ContainerStarted","Data":"2338fdbcf0fa0c2028733c780ecea88b1667548031c5ca23778f2b14ea09c78f"}
Oct 30 03:27:31.991729 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:31.991211    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6" event={"ID":"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14","Type":"ContainerStarted","Data":"0e0439d87f2fedd6d370b2c6905e92c9ac78fc5c7756fb8cbf412b648d3083fe"}
Oct 30 03:27:31.993042 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:31.993011    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz" event={"ID":"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516","Type":"ContainerStarted","Data":"147cff012941a59e39528b023662581efc880e1e18f6495a7f8c10a7879a30b6"}
Oct 30 03:27:31.993174 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:31.993069    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz" event={"ID":"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516","Type":"ContainerStarted","Data":"5789aaea6fb461da992a8ef08f65c87694f733f74804bb3a7b4d61144db2bd45"}
Oct 30 03:27:31.994821 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:31.994798    4278 generic.go:356] "Generic (PLEG): container finished" podID="484aa77b-b879-4fe5-acad-4f1e1d0787d0" containerID="8578c96c8ddca954cb7ba677a0dbb9696b978f315ffc6f061ff10eb4d1038dcd" exitCode=0
Oct 30 03:27:31.994915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:31.994846    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q" event={"ID":"484aa77b-b879-4fe5-acad-4f1e1d0787d0","Type":"ContainerDied","Data":"8578c96c8ddca954cb7ba677a0dbb9696b978f315ffc6f061ff10eb4d1038dcd"}
Oct 30 03:27:31.994915 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:31.994889    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q" event={"ID":"484aa77b-b879-4fe5-acad-4f1e1d0787d0","Type":"ContainerStarted","Data":"a5e3d8dd34c2f4f125024cbab6b592b9371777d73d5c0680875590f98b9dbc15"}
Oct 30 03:27:32.029775 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:32.029690    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6" podStartSLOduration=2.029672259 podStartE2EDuration="2.029672259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:32.028054445 +0000 UTC m=+2789.952262632" watchObservedRunningTime="2026-10-30 03:27:32.029672259 +0000 UTC m=+2789.953880446"
Oct 30 03:27:32.064855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:32.064799    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz" podStartSLOduration=2.064782389 podStartE2EDuration="2.064782389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:32.06380461 +0000 UTC m=+2789.988012798" watchObservedRunningTime="2026-10-30 03:27:32.064782389 +0000 UTC m=+2789.988990576"
Oct 30 03:27:33.006585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.006538    4278 generic.go:356] "Generic (PLEG): container finished" podID="4f43f0a6-77de-47a5-9bdb-4b4763f6eb14" containerID="2338fdbcf0fa0c2028733c780ecea88b1667548031c5ca23778f2b14ea09c78f" exitCode=0
Oct 30 03:27:33.007121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.006621    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6" event={"ID":"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14","Type":"ContainerDied","Data":"2338fdbcf0fa0c2028733c780ecea88b1667548031c5ca23778f2b14ea09c78f"}
Oct 30 03:27:33.008868 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.008842    4278 generic.go:356] "Generic (PLEG): container finished" podID="f1086f8f-0dde-493c-8fd2-c8fe6e2ca516" containerID="147cff012941a59e39528b023662581efc880e1e18f6495a7f8c10a7879a30b6" exitCode=0
Oct 30 03:27:33.009029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.008935    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz" event={"ID":"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516","Type":"ContainerDied","Data":"147cff012941a59e39528b023662581efc880e1e18f6495a7f8c10a7879a30b6"}
Oct 30 03:27:33.169480 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.169445    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:33.277874 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.277782    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484aa77b-b879-4fe5-acad-4f1e1d0787d0-kube-api-access-7nznm\" (UniqueName: \"kubernetes.io/projected/484aa77b-b879-4fe5-acad-4f1e1d0787d0-kube-api-access-7nznm\") pod \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") "
Oct 30 03:27:33.278043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.278007    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-xks-ca\") pod \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") "
Oct 30 03:27:33.278087 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.278059    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-openshift-ca\") pod \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\" (UID: \"484aa77b-b879-4fe5-acad-4f1e1d0787d0\") "
Oct 30 03:27:33.278465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.278413    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-openshift-ca" pod "484aa77b-b879-4fe5-acad-4f1e1d0787d0" (UID: "484aa77b-b879-4fe5-acad-4f1e1d0787d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:33.279972 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.279945    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484aa77b-b879-4fe5-acad-4f1e1d0787d0-kube-api-access-7nznm" pod "484aa77b-b879-4fe5-acad-4f1e1d0787d0" (UID: "484aa77b-b879-4fe5-acad-4f1e1d0787d0"). InnerVolumeSpecName "kube-api-access-7nznm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:33.280082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.280071    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-xks-ca" pod "484aa77b-b879-4fe5-acad-4f1e1d0787d0" (UID: "484aa77b-b879-4fe5-acad-4f1e1d0787d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:33.380017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.379983    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:33.380017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.380013    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/484aa77b-b879-4fe5-acad-4f1e1d0787d0-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:33.380017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:33.380022    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nznm\" (UniqueName: \"kubernetes.io/projected/484aa77b-b879-4fe5-acad-4f1e1d0787d0-kube-api-access-7nznm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:34.017408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.017376    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"
Oct 30 03:27:34.017408 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.017401    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q" event={"ID":"484aa77b-b879-4fe5-acad-4f1e1d0787d0","Type":"ContainerDied","Data":"a5e3d8dd34c2f4f125024cbab6b592b9371777d73d5c0680875590f98b9dbc15"}
Oct 30 03:27:34.017995 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.017479    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e3d8dd34c2f4f125024cbab6b592b9371777d73d5c0680875590f98b9dbc15"
Oct 30 03:27:34.028495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.028462    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"]
Oct 30 03:27:34.039463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.039417    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04334378-vp4vc"]
Oct 30 03:27:34.094780 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.094741    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"]
Oct 30 03:27:34.095017 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.094990    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy" containerID="cri-o://74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c" gracePeriod=30
Oct 30 03:27:34.190837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.190810    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:34.199338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.198875    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6136e008-a09d-498c-a9fe-275b2f9b6f4b" path="/var/lib/kubelet/pods/6136e008-a09d-498c-a9fe-275b2f9b6f4b/volumes"
Oct 30 03:27:34.223151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.223121    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:34.290822 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.290730    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-kube-api-access-dxg85\" (UniqueName: \"kubernetes.io/projected/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-kube-api-access-dxg85\") pod \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") "
Oct 30 03:27:34.291080 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.290838    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-openshift-ca\") pod \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") "
Oct 30 03:27:34.291080 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.290869    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-xks-ca\") pod \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\" (UID: \"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516\") "
Oct 30 03:27:34.291313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.291283    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-openshift-ca" pod "f1086f8f-0dde-493c-8fd2-c8fe6e2ca516" (UID: "f1086f8f-0dde-493c-8fd2-c8fe6e2ca516"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:34.293140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.293108    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-xks-ca" pod "f1086f8f-0dde-493c-8fd2-c8fe6e2ca516" (UID: "f1086f8f-0dde-493c-8fd2-c8fe6e2ca516"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:34.293531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.293507    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-kube-api-access-dxg85" pod "f1086f8f-0dde-493c-8fd2-c8fe6e2ca516" (UID: "f1086f8f-0dde-493c-8fd2-c8fe6e2ca516"). InnerVolumeSpecName "kube-api-access-dxg85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:34.392671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.392637    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-xks-ca\") pod \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") "
Oct 30 03:27:34.392863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.392845    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-openshift-ca\") pod \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") "
Oct 30 03:27:34.393035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.393012    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-kube-api-access-rzhlf\" (UniqueName: \"kubernetes.io/projected/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-kube-api-access-rzhlf\") pod \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\" (UID: \"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14\") "
Oct 30 03:27:34.393209 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.393183    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-openshift-ca" pod "4f43f0a6-77de-47a5-9bdb-4b4763f6eb14" (UID: "4f43f0a6-77de-47a5-9bdb-4b4763f6eb14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:34.393534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.393508    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxg85\" (UniqueName: \"kubernetes.io/projected/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-kube-api-access-dxg85\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:34.393534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.393536    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:34.393724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.393547    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:34.393724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.393556    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:34.395074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.395046    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-xks-ca" pod "4f43f0a6-77de-47a5-9bdb-4b4763f6eb14" (UID: "4f43f0a6-77de-47a5-9bdb-4b4763f6eb14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:34.395172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.395137    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-kube-api-access-rzhlf" pod "4f43f0a6-77de-47a5-9bdb-4b4763f6eb14" (UID: "4f43f0a6-77de-47a5-9bdb-4b4763f6eb14"). InnerVolumeSpecName "kube-api-access-rzhlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:34.505883 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.504330    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzhlf\" (UniqueName: \"kubernetes.io/projected/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-kube-api-access-rzhlf\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:34.505883 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:34.504369    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:35.027116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:35.027079    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"
Oct 30 03:27:35.027116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:35.027107    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz" event={"ID":"f1086f8f-0dde-493c-8fd2-c8fe6e2ca516","Type":"ContainerDied","Data":"5789aaea6fb461da992a8ef08f65c87694f733f74804bb3a7b4d61144db2bd45"}
Oct 30 03:27:35.027698 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:35.027158    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5789aaea6fb461da992a8ef08f65c87694f733f74804bb3a7b4d61144db2bd45"
Oct 30 03:27:35.028938 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:35.028908    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"
Oct 30 03:27:35.029082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:35.028902    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6" event={"ID":"4f43f0a6-77de-47a5-9bdb-4b4763f6eb14","Type":"ContainerDied","Data":"0e0439d87f2fedd6d370b2c6905e92c9ac78fc5c7756fb8cbf412b648d3083fe"}
Oct 30 03:27:35.029082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:35.029039    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e0439d87f2fedd6d370b2c6905e92c9ac78fc5c7756fb8cbf412b648d3083fe"
Oct 30 03:27:35.621576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:35.621530    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:27:35.621765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:35.621603    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:27:38.927140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:38.927094    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main" containerID="cri-o://da388f186cfbe9123891acc9cc64c6b56b5179d7b5d9d65926b0f035f93ac532" gracePeriod=45
Oct 30 03:27:38.950361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:38.950324    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" containerID="cri-o://561e1cc7b9ebbd1c89ce4d20c98ebdd374d3a91f668bef511159abf9c2dfdf5e" gracePeriod=45
Oct 30 03:27:39.069566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.069528    4278 generic.go:356] "Generic (PLEG): container finished" podID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerID="561e1cc7b9ebbd1c89ce4d20c98ebdd374d3a91f668bef511159abf9c2dfdf5e" exitCode=0
Oct 30 03:27:39.069717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.069598    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" event={"ID":"6c2f9269-eb24-4bb1-b784-da668f2e768f","Type":"ContainerDied","Data":"561e1cc7b9ebbd1c89ce4d20c98ebdd374d3a91f668bef511159abf9c2dfdf5e"}
Oct 30 03:27:39.071710 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.071685    4278 generic.go:356] "Generic (PLEG): container finished" podID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerID="da388f186cfbe9123891acc9cc64c6b56b5179d7b5d9d65926b0f035f93ac532" exitCode=0
Oct 30 03:27:39.071840 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.071784    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" event={"ID":"764d1f39-ea66-4aa4-99ce-59cd2cda80b5","Type":"ContainerDied","Data":"da388f186cfbe9123891acc9cc64c6b56b5179d7b5d9d65926b0f035f93ac532"}
Oct 30 03:27:39.245491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.245407    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 03:27:39.254300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.254273    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 03:27:39.367346 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367309    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-model-cache\") pod \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") "
Oct 30 03:27:39.367589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367353    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c2f9269-eb24-4bb1-b784-da668f2e768f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2f9269-eb24-4bb1-b784-da668f2e768f-tls-certs\") pod \"6c2f9269-eb24-4bb1-b784-da668f2e768f\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") "
Oct 30 03:27:39.367589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367406    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-tmp-dir\") pod \"6c2f9269-eb24-4bb1-b784-da668f2e768f\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") "
Oct 30 03:27:39.367589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367462    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tls-certs\") pod \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") "
Oct 30 03:27:39.367589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367483    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-model-cache\") pod \"6c2f9269-eb24-4bb1-b784-da668f2e768f\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") "
Oct 30 03:27:39.367589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367519    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-kube-api-access-cp9xg\" (UniqueName: \"kubernetes.io/projected/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-kube-api-access-cp9xg\") pod \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") "
Oct 30 03:27:39.367589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367543    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tmp-dir\") pod \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") "
Oct 30 03:27:39.367589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367566    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-dshm\") pod \"6c2f9269-eb24-4bb1-b784-da668f2e768f\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") "
Oct 30 03:27:39.367992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367620    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-home\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-home\") pod \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") "
Oct 30 03:27:39.367992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367693    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-dshm\") pod \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\" (UID: \"764d1f39-ea66-4aa4-99ce-59cd2cda80b5\") "
Oct 30 03:27:39.367992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367705    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-model-cache" pod "764d1f39-ea66-4aa4-99ce-59cd2cda80b5" (UID: "764d1f39-ea66-4aa4-99ce-59cd2cda80b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.367992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367832    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2f9269-eb24-4bb1-b784-da668f2e768f-kube-api-access-ww864\" (UniqueName: \"kubernetes.io/projected/6c2f9269-eb24-4bb1-b784-da668f2e768f-kube-api-access-ww864\") pod \"6c2f9269-eb24-4bb1-b784-da668f2e768f\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") "
Oct 30 03:27:39.367992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.367916    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-home\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-home\") pod \"6c2f9269-eb24-4bb1-b784-da668f2e768f\" (UID: \"6c2f9269-eb24-4bb1-b784-da668f2e768f\") "
Oct 30 03:27:39.368649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.368621    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.369126 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.368912    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tmp-dir" pod "764d1f39-ea66-4aa4-99ce-59cd2cda80b5" (UID: "764d1f39-ea66-4aa4-99ce-59cd2cda80b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.369458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.369396    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-tmp-dir" pod "6c2f9269-eb24-4bb1-b784-da668f2e768f" (UID: "6c2f9269-eb24-4bb1-b784-da668f2e768f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.369458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.369434    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-home" pod "6c2f9269-eb24-4bb1-b784-da668f2e768f" (UID: "6c2f9269-eb24-4bb1-b784-da668f2e768f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.369620 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.369598    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-model-cache" pod "6c2f9269-eb24-4bb1-b784-da668f2e768f" (UID: "6c2f9269-eb24-4bb1-b784-da668f2e768f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.369692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.369664    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-home" pod "764d1f39-ea66-4aa4-99ce-59cd2cda80b5" (UID: "764d1f39-ea66-4aa4-99ce-59cd2cda80b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.371634 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.371608    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-dshm" pod "764d1f39-ea66-4aa4-99ce-59cd2cda80b5" (UID: "764d1f39-ea66-4aa4-99ce-59cd2cda80b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.371777 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.371752    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2f9269-eb24-4bb1-b784-da668f2e768f-kube-api-access-ww864" pod "6c2f9269-eb24-4bb1-b784-da668f2e768f" (UID: "6c2f9269-eb24-4bb1-b784-da668f2e768f"). InnerVolumeSpecName "kube-api-access-ww864". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:39.372281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.372245    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tls-certs" pod "764d1f39-ea66-4aa4-99ce-59cd2cda80b5" (UID: "764d1f39-ea66-4aa4-99ce-59cd2cda80b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:39.372401 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.372354    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2f9269-eb24-4bb1-b784-da668f2e768f-tls-certs" pod "6c2f9269-eb24-4bb1-b784-da668f2e768f" (UID: "6c2f9269-eb24-4bb1-b784-da668f2e768f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:39.372731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.372706    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-dshm" pod "6c2f9269-eb24-4bb1-b784-da668f2e768f" (UID: "6c2f9269-eb24-4bb1-b784-da668f2e768f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.373416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.373394    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-kube-api-access-cp9xg" pod "764d1f39-ea66-4aa4-99ce-59cd2cda80b5" (UID: "764d1f39-ea66-4aa4-99ce-59cd2cda80b5"). InnerVolumeSpecName "kube-api-access-cp9xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:39.397618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.397587    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 03:27:39.471889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.471776    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-token\" (UniqueName: \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-token\") pod \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") "
Oct 30 03:27:39.471889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.471843    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-kube-api-access-jz5bh\" (UniqueName: \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-kube-api-access-jz5bh\") pod \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") "
Oct 30 03:27:39.472137 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.471894    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-certs\") pod \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") "
Oct 30 03:27:39.472137 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.471937    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-credential-socket\") pod \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") "
Oct 30 03:27:39.472137 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.472049    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istiod-ca-cert\") pod \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") "
Oct 30 03:27:39.472137 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.472116    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-envoy\") pod \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") "
Oct 30 03:27:39.472343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.472156    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-data\") pod \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") "
Oct 30 03:27:39.472343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.472193    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-podinfo\") pod \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") "
Oct 30 03:27:39.472343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.472230    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-socket\") pod \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\" (UID: \"6760b8c9-1381-4a45-91a7-3c8b4d67b315\") "
Oct 30 03:27:39.474815 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.474774    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-certs" pod "6760b8c9-1381-4a45-91a7-3c8b4d67b315" (UID: "6760b8c9-1381-4a45-91a7-3c8b4d67b315"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.475206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.475174    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istiod-ca-cert" pod "6760b8c9-1381-4a45-91a7-3c8b4d67b315" (UID: "6760b8c9-1381-4a45-91a7-3c8b4d67b315"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:39.475962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.475909    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-socket" pod "6760b8c9-1381-4a45-91a7-3c8b4d67b315" (UID: "6760b8c9-1381-4a45-91a7-3c8b4d67b315"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.479416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.479197    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-data" pod "6760b8c9-1381-4a45-91a7-3c8b4d67b315" (UID: "6760b8c9-1381-4a45-91a7-3c8b4d67b315"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.479750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.479617    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-credential-socket" pod "6760b8c9-1381-4a45-91a7-3c8b4d67b315" (UID: "6760b8c9-1381-4a45-91a7-3c8b4d67b315"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.479750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.479728    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-token" pod "6760b8c9-1381-4a45-91a7-3c8b4d67b315" (UID: "6760b8c9-1381-4a45-91a7-3c8b4d67b315"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:39.480323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480275    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480306    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2f9269-eb24-4bb1-b784-da668f2e768f-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480320    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480332    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480345    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480358    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480370    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480384    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp9xg\" (UniqueName: \"kubernetes.io/projected/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-kube-api-access-cp9xg\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480398    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480411    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480439    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480452    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480463    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764d1f39-ea66-4aa4-99ce-59cd2cda80b5-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480475    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480485    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480498    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww864\" (UniqueName: \"kubernetes.io/projected/6c2f9269-eb24-4bb1-b784-da668f2e768f-kube-api-access-ww864\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.480587 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.480510    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2f9269-eb24-4bb1-b784-da668f2e768f-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.484630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.484600    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-envoy" pod "6760b8c9-1381-4a45-91a7-3c8b4d67b315" (UID: "6760b8c9-1381-4a45-91a7-3c8b4d67b315"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.485705 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.485658    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-podinfo" pod "6760b8c9-1381-4a45-91a7-3c8b4d67b315" (UID: "6760b8c9-1381-4a45-91a7-3c8b4d67b315"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:27:39.489841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.489814    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-kube-api-access-jz5bh" pod "6760b8c9-1381-4a45-91a7-3c8b4d67b315" (UID: "6760b8c9-1381-4a45-91a7-3c8b4d67b315"). InnerVolumeSpecName "kube-api-access-jz5bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:39.582020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.581979    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz5bh\" (UniqueName: \"kubernetes.io/projected/6760b8c9-1381-4a45-91a7-3c8b4d67b315-kube-api-access-jz5bh\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.582020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.582014    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.582020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:39.582026    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6760b8c9-1381-4a45-91a7-3c8b4d67b315-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:40.079883 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.079848    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"
Oct 30 03:27:40.080355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.079878    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" event={"ID":"6c2f9269-eb24-4bb1-b784-da668f2e768f","Type":"ContainerDied","Data":"8e3d456a604b358f1312a7dcb186ad2316486cec5c805a326e0918a4a9f5aec3"}
Oct 30 03:27:40.080355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.079927    4278 scope.go:122] "RemoveContainer" containerID="561e1cc7b9ebbd1c89ce4d20c98ebdd374d3a91f668bef511159abf9c2dfdf5e"
Oct 30 03:27:40.081688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.081515    4278 generic.go:356] "Generic (PLEG): container finished" podID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerID="74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c" exitCode=0
Oct 30 03:27:40.081688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.081571    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"
Oct 30 03:27:40.081688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.081591    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" event={"ID":"6760b8c9-1381-4a45-91a7-3c8b4d67b315","Type":"ContainerDied","Data":"74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c"}
Oct 30 03:27:40.081688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.081650    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv" event={"ID":"6760b8c9-1381-4a45-91a7-3c8b4d67b315","Type":"ContainerDied","Data":"1de74c1f0996329062128ad7986c66f3ab754a2a1325c2ad321493b9d0f64c56"}
Oct 30 03:27:40.083584 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.083562    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"
Oct 30 03:27:40.083698 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.083560    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp" event={"ID":"764d1f39-ea66-4aa4-99ce-59cd2cda80b5","Type":"ContainerDied","Data":"f5310163b4ef9faeb4ccdbdc53417a1a3e163304fe3ab7df80c54cb1b4033e31"}
Oct 30 03:27:40.099360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.099333    4278 scope.go:122] "RemoveContainer" containerID="74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c"
Oct 30 03:27:40.110001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.109979    4278 scope.go:122] "RemoveContainer" containerID="74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c"
Oct 30 03:27:40.110311 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:27:40.110291    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c\": container with ID starting with 74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c not found: ID does not exist" containerID="74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c"
Oct 30 03:27:40.110358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.110324    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c"} err="failed to get container status \"74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c\": rpc error: code = NotFound desc = could not find container \"74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c\": container with ID starting with 74eb53bccd77656e18e7aad6a62ce81610f63b6de5449ad9aaea32927cec588c not found: ID does not exist"
Oct 30 03:27:40.110358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.110343    4278 scope.go:122] "RemoveContainer" containerID="da388f186cfbe9123891acc9cc64c6b56b5179d7b5d9d65926b0f035f93ac532"
Oct 30 03:27:40.158687 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.158627    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:27:40.160533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.160495    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"]
Oct 30 03:27:40.164514 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.164487    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fp9dv"]
Oct 30 03:27:40.199867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.199793    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" path="/var/lib/kubelet/pods/6760b8c9-1381-4a45-91a7-3c8b4d67b315/volumes"
Oct 30 03:27:40.200124 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.200110    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"]
Oct 30 03:27:40.208607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.208574    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/simulator-w3-d9f3ac-kserve-6d46cf55f8-cm7c7"]
Oct 30 03:27:40.241558 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.241514    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"]
Oct 30 03:27:40.244599 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.244570    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-2-w3-d9f3ac-kserve-6465b67885-ndqxp"]
Oct 30 03:27:40.768683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.768644    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"]
Oct 30 03:27:40.769211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769195    4278 cpu_manager.go:414] "containerMap: removing container" podUID="484aa77b-b879-4fe5-acad-4f1e1d0787d0" containerName="revoke-keys"
Oct 30 03:27:40.769288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769212    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484aa77b-b879-4fe5-acad-4f1e1d0787d0" containerName="revoke-keys"
Oct 30 03:27:40.769288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769224    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f1086f8f-0dde-493c-8fd2-c8fe6e2ca516" containerName="revoke-keys"
Oct 30 03:27:40.769288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769229    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1086f8f-0dde-493c-8fd2-c8fe6e2ca516" containerName="revoke-keys"
Oct 30 03:27:40.769288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769258    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main"
Oct 30 03:27:40.769288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769264    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main"
Oct 30 03:27:40.769288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769271    4278 cpu_manager.go:414] "containerMap: removing container" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main"
Oct 30 03:27:40.769288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769276    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main"
Oct 30 03:27:40.769288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769284    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy"
Oct 30 03:27:40.769288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769288    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy"
Oct 30 03:27:40.769656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769316    4278 cpu_manager.go:414] "containerMap: removing container" podUID="4f43f0a6-77de-47a5-9bdb-4b4763f6eb14" containerName="revoke-keys"
Oct 30 03:27:40.769656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769322    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f43f0a6-77de-47a5-9bdb-4b4763f6eb14" containerName="revoke-keys"
Oct 30 03:27:40.769656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769398    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f43f0a6-77de-47a5-9bdb-4b4763f6eb14" containerName="revoke-keys"
Oct 30 03:27:40.769656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769413    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="484aa77b-b879-4fe5-acad-4f1e1d0787d0" containerName="revoke-keys"
Oct 30 03:27:40.769656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769433    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1086f8f-0dde-493c-8fd2-c8fe6e2ca516" containerName="revoke-keys"
Oct 30 03:27:40.769656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769448    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" containerName="main"
Oct 30 03:27:40.769656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769455    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" containerName="main"
Oct 30 03:27:40.769656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.769464    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6760b8c9-1381-4a45-91a7-3c8b4d67b315" containerName="istio-proxy"
Oct 30 03:27:40.774355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.774331    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:40.792168 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.792133    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"]
Oct 30 03:27:40.795023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.794995    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a123204f-sjjkd\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:40.795206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.795189    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swn44\" (UniqueName: \"kubernetes.io/projected/42229be7-6a18-45e5-82a2-7a2f3f173836-kube-api-access-swn44\") pod \"maas-api-revoke-sub-a123204f-sjjkd\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:40.795270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.795223    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a123204f-sjjkd\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:40.799486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.799455    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:27:40.802650 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.802626    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:27:40.897945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.897903    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swn44\" (UniqueName: \"kubernetes.io/projected/42229be7-6a18-45e5-82a2-7a2f3f173836-kube-api-access-swn44\") pod \"maas-api-revoke-sub-a123204f-sjjkd\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:40.898615 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.898582    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a123204f-sjjkd\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:40.899386 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.899360    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a123204f-sjjkd\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:40.900531 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.900491    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a123204f-sjjkd\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:40.924492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.920725    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swn44\" (UniqueName: \"kubernetes.io/projected/42229be7-6a18-45e5-82a2-7a2f3f173836-kube-api-access-swn44\") pod \"maas-api-revoke-sub-a123204f-sjjkd\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:40.929053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:40.929012    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a123204f-sjjkd\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:41.088302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:41.088196    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:41.390323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:41.390298    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"]
Oct 30 03:27:41.392910 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:27:41.392859    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42229be7_6a18_45e5_82a2_7a2f3f173836.slice/crio-d0ff37e7fa4a7b8a79be7cdc88b7bbc1a3d3b79ee7fdcb0d3ae09ae81b4f5a98 WatchSource:0}: Error finding container d0ff37e7fa4a7b8a79be7cdc88b7bbc1a3d3b79ee7fdcb0d3ae09ae81b4f5a98: Status 404 returned error can't find the container with id d0ff37e7fa4a7b8a79be7cdc88b7bbc1a3d3b79ee7fdcb0d3ae09ae81b4f5a98
Oct 30 03:27:42.104482 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:42.104436    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd" event={"ID":"42229be7-6a18-45e5-82a2-7a2f3f173836","Type":"ContainerStarted","Data":"d25910361a30750827f2655417dd20ac61bf1621a0494709a9fee04a6b897958"}
Oct 30 03:27:42.104933 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:42.104510    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd" event={"ID":"42229be7-6a18-45e5-82a2-7a2f3f173836","Type":"ContainerStarted","Data":"d0ff37e7fa4a7b8a79be7cdc88b7bbc1a3d3b79ee7fdcb0d3ae09ae81b4f5a98"}
Oct 30 03:27:42.140795 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:42.140736    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd" podStartSLOduration=2.140716107 podStartE2EDuration="2.140716107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:42.140097947 +0000 UTC m=+2800.064306136" watchObservedRunningTime="2026-10-30 03:27:42.140716107 +0000 UTC m=+2800.064924298"
Oct 30 03:27:42.199340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:42.199293    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2f9269-eb24-4bb1-b784-da668f2e768f" path="/var/lib/kubelet/pods/6c2f9269-eb24-4bb1-b784-da668f2e768f/volumes"
Oct 30 03:27:42.199924 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:42.199899    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764d1f39-ea66-4aa4-99ce-59cd2cda80b5" path="/var/lib/kubelet/pods/764d1f39-ea66-4aa4-99ce-59cd2cda80b5/volumes"
Oct 30 03:27:43.112746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:43.112707    4278 generic.go:356] "Generic (PLEG): container finished" podID="42229be7-6a18-45e5-82a2-7a2f3f173836" containerID="d25910361a30750827f2655417dd20ac61bf1621a0494709a9fee04a6b897958" exitCode=0
Oct 30 03:27:43.113253 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:43.112790    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd" event={"ID":"42229be7-6a18-45e5-82a2-7a2f3f173836","Type":"ContainerDied","Data":"d25910361a30750827f2655417dd20ac61bf1621a0494709a9fee04a6b897958"}
Oct 30 03:27:44.283625 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:44.283598    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:44.341323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:44.341288    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-xks-ca\") pod \"42229be7-6a18-45e5-82a2-7a2f3f173836\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") "
Oct 30 03:27:44.341551 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:44.341391    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42229be7-6a18-45e5-82a2-7a2f3f173836-kube-api-access-swn44\" (UniqueName: \"kubernetes.io/projected/42229be7-6a18-45e5-82a2-7a2f3f173836-kube-api-access-swn44\") pod \"42229be7-6a18-45e5-82a2-7a2f3f173836\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") "
Oct 30 03:27:44.341621 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:44.341556    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-openshift-ca\") pod \"42229be7-6a18-45e5-82a2-7a2f3f173836\" (UID: \"42229be7-6a18-45e5-82a2-7a2f3f173836\") "
Oct 30 03:27:44.341991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:44.341952    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-openshift-ca" pod "42229be7-6a18-45e5-82a2-7a2f3f173836" (UID: "42229be7-6a18-45e5-82a2-7a2f3f173836"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:44.344167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:44.343310    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:44.349462 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:44.344433    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-xks-ca" pod "42229be7-6a18-45e5-82a2-7a2f3f173836" (UID: "42229be7-6a18-45e5-82a2-7a2f3f173836"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:44.354757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:44.354709    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42229be7-6a18-45e5-82a2-7a2f3f173836-kube-api-access-swn44" pod "42229be7-6a18-45e5-82a2-7a2f3f173836" (UID: "42229be7-6a18-45e5-82a2-7a2f3f173836"). InnerVolumeSpecName "kube-api-access-swn44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:44.444146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:44.444047    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42229be7-6a18-45e5-82a2-7a2f3f173836-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:44.444146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:44.444108    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swn44\" (UniqueName: \"kubernetes.io/projected/42229be7-6a18-45e5-82a2-7a2f3f173836-kube-api-access-swn44\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:45.128855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:45.128814    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd" event={"ID":"42229be7-6a18-45e5-82a2-7a2f3f173836","Type":"ContainerDied","Data":"d0ff37e7fa4a7b8a79be7cdc88b7bbc1a3d3b79ee7fdcb0d3ae09ae81b4f5a98"}
Oct 30 03:27:45.129098 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:45.128912    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0ff37e7fa4a7b8a79be7cdc88b7bbc1a3d3b79ee7fdcb0d3ae09ae81b4f5a98"
Oct 30 03:27:45.129098 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:45.128828    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"
Oct 30 03:27:46.108532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:46.108493    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"]
Oct 30 03:27:46.119198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:46.119159    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9d3552-g6hw8"]
Oct 30 03:27:46.198929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:46.198892    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfe0574-64e3-45bc-a097-ee984c674fa7" path="/var/lib/kubelet/pods/1cfe0574-64e3-45bc-a097-ee984c674fa7/volumes"
Oct 30 03:27:53.444888 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:53.444842    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"]
Oct 30 03:27:53.445514 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:53.445151    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy" containerID="cri-o://86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60" gracePeriod=30
Oct 30 03:27:54.190401 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:27:54.190358    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:27:54.268417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.268377    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"]
Oct 30 03:27:54.269023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.268999    4278 cpu_manager.go:414] "containerMap: removing container" podUID="42229be7-6a18-45e5-82a2-7a2f3f173836" containerName="revoke-keys"
Oct 30 03:27:54.269131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.269024    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42229be7-6a18-45e5-82a2-7a2f3f173836" containerName="revoke-keys"
Oct 30 03:27:54.269206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.269193    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="42229be7-6a18-45e5-82a2-7a2f3f173836" containerName="revoke-keys"
Oct 30 03:27:54.273897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.273872    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.302516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.302470    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:27:54.302516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.302509    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:27:54.305898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.305865    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"]
Oct 30 03:27:54.404283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.404237    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.404538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.404487    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7jzv\" (UniqueName: \"kubernetes.io/projected/288dc87e-bf37-47bc-9f9f-5e500e6c1308-kube-api-access-x7jzv\") pod \"maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.404661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.404639    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.505746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.505645    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7jzv\" (UniqueName: \"kubernetes.io/projected/288dc87e-bf37-47bc-9f9f-5e500e6c1308-kube-api-access-x7jzv\") pod \"maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.506280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.505748    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.506280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.505809    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.506417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.506396    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.508286 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.508261    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.524711 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.524682    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7jzv\" (UniqueName: \"kubernetes.io/projected/288dc87e-bf37-47bc-9f9f-5e500e6c1308-kube-api-access-x7jzv\") pod \"maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.588086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.588054    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:54.892075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:54.892043    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"]
Oct 30 03:27:54.894976 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:27:54.894940    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288dc87e_bf37_47bc_9f9f_5e500e6c1308.slice/crio-3ee494136b831ed2a5a4d01678777b0d95362a6dfa5fb10b3c2dd39dc6d9198e WatchSource:0}: Error finding container 3ee494136b831ed2a5a4d01678777b0d95362a6dfa5fb10b3c2dd39dc6d9198e: Status 404 returned error can't find the container with id 3ee494136b831ed2a5a4d01678777b0d95362a6dfa5fb10b3c2dd39dc6d9198e
Oct 30 03:27:55.202928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:55.202888    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p" event={"ID":"288dc87e-bf37-47bc-9f9f-5e500e6c1308","Type":"ContainerStarted","Data":"3ee494136b831ed2a5a4d01678777b0d95362a6dfa5fb10b3c2dd39dc6d9198e"}
Oct 30 03:27:56.211365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:56.211262    4278 generic.go:356] "Generic (PLEG): container finished" podID="288dc87e-bf37-47bc-9f9f-5e500e6c1308" containerID="4d575e34f47529b6dda15c2d0239ebc09ca8519d38544a72f146b6fdd07bbc8d" exitCode=0
Oct 30 03:27:56.211830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:56.211359    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p" event={"ID":"288dc87e-bf37-47bc-9f9f-5e500e6c1308","Type":"ContainerDied","Data":"4d575e34f47529b6dda15c2d0239ebc09ca8519d38544a72f146b6fdd07bbc8d"}
Oct 30 03:27:57.400098 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.400058    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:57.567768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.567733    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-openshift-ca\") pod \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") "
Oct 30 03:27:57.568020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.567836    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288dc87e-bf37-47bc-9f9f-5e500e6c1308-kube-api-access-x7jzv\" (UniqueName: \"kubernetes.io/projected/288dc87e-bf37-47bc-9f9f-5e500e6c1308-kube-api-access-x7jzv\") pod \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") "
Oct 30 03:27:57.568020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.567995    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-xks-ca\") pod \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\" (UID: \"288dc87e-bf37-47bc-9f9f-5e500e6c1308\") "
Oct 30 03:27:57.568253 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.568227    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-openshift-ca" pod "288dc87e-bf37-47bc-9f9f-5e500e6c1308" (UID: "288dc87e-bf37-47bc-9f9f-5e500e6c1308"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:57.568777 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.568633    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:57.570764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.570719    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-xks-ca" pod "288dc87e-bf37-47bc-9f9f-5e500e6c1308" (UID: "288dc87e-bf37-47bc-9f9f-5e500e6c1308"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:57.571754 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.571688    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288dc87e-bf37-47bc-9f9f-5e500e6c1308-kube-api-access-x7jzv" pod "288dc87e-bf37-47bc-9f9f-5e500e6c1308" (UID: "288dc87e-bf37-47bc-9f9f-5e500e6c1308"). InnerVolumeSpecName "kube-api-access-x7jzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:57.573393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.573351    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"]
Oct 30 03:27:57.574208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.574183    4278 cpu_manager.go:414] "containerMap: removing container" podUID="288dc87e-bf37-47bc-9f9f-5e500e6c1308" containerName="revoke-keys"
Oct 30 03:27:57.574208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.574208    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288dc87e-bf37-47bc-9f9f-5e500e6c1308" containerName="revoke-keys"
Oct 30 03:27:57.574455 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.574407    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="288dc87e-bf37-47bc-9f9f-5e500e6c1308" containerName="revoke-keys"
Oct 30 03:27:57.578001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.577981    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:57.602384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.602040    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"]
Oct 30 03:27:57.671469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.670055    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7jzv\" (UniqueName: \"kubernetes.io/projected/288dc87e-bf37-47bc-9f9f-5e500e6c1308-kube-api-access-x7jzv\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:57.671469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.670096    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288dc87e-bf37-47bc-9f9f-5e500e6c1308-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:57.770750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.770710    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:57.770975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.770845    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:57.771045 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.771013    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpfs\" (UniqueName: \"kubernetes.io/projected/95c27bd7-2869-4e56-9b87-a64522f6284e-kube-api-access-btpfs\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:57.873148 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.873054    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpfs\" (UniqueName: \"kubernetes.io/projected/95c27bd7-2869-4e56-9b87-a64522f6284e-kube-api-access-btpfs\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:57.873344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.873155    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:57.873344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.873205    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:57.873928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.873904    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:57.875692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.875665    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:57.891832 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.891802    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpfs\" (UniqueName: \"kubernetes.io/projected/95c27bd7-2869-4e56-9b87-a64522f6284e-kube-api-access-btpfs\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:57.911823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:57.911794    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:27:58.227521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.227482    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"
Oct 30 03:27:58.227720 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.227534    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p" event={"ID":"288dc87e-bf37-47bc-9f9f-5e500e6c1308","Type":"ContainerDied","Data":"3ee494136b831ed2a5a4d01678777b0d95362a6dfa5fb10b3c2dd39dc6d9198e"}
Oct 30 03:27:58.227720 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.227596    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ee494136b831ed2a5a4d01678777b0d95362a6dfa5fb10b3c2dd39dc6d9198e"
Oct 30 03:27:58.232101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.232079    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"]
Oct 30 03:27:58.234554 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:27:58.234528    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c27bd7_2869_4e56_9b87_a64522f6284e.slice/crio-e44d735adca9b548cad32b9177f4d84693d4223cecd6affa406e6241940ab515 WatchSource:0}: Error finding container e44d735adca9b548cad32b9177f4d84693d4223cecd6affa406e6241940ab515: Status 404 returned error can't find the container with id e44d735adca9b548cad32b9177f4d84693d4223cecd6affa406e6241940ab515
Oct 30 03:27:58.730465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.730384    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:27:58.782564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.782527    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-istio-token\" (UniqueName: \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-istio-token\") pod \"ffac99be-3709-4c4f-9526-7c290027b026\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") "
Oct 30 03:27:58.782745 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.782609    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-kube-api-access-92zgz\" (UniqueName: \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-kube-api-access-92zgz\") pod \"ffac99be-3709-4c4f-9526-7c290027b026\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") "
Oct 30 03:27:58.782745 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.782651    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-certs\") pod \"ffac99be-3709-4c4f-9526-7c290027b026\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") "
Oct 30 03:27:58.782745 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.782727    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffac99be-3709-4c4f-9526-7c290027b026-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffac99be-3709-4c4f-9526-7c290027b026-istio-podinfo\") pod \"ffac99be-3709-4c4f-9526-7c290027b026\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") "
Oct 30 03:27:58.782926 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.782807    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-data\") pod \"ffac99be-3709-4c4f-9526-7c290027b026\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") "
Oct 30 03:27:58.782926 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.782897    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-socket\") pod \"ffac99be-3709-4c4f-9526-7c290027b026\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") "
Oct 30 03:27:58.783039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.782927    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-envoy\") pod \"ffac99be-3709-4c4f-9526-7c290027b026\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") "
Oct 30 03:27:58.783039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.782959    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffac99be-3709-4c4f-9526-7c290027b026-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffac99be-3709-4c4f-9526-7c290027b026-istiod-ca-cert\") pod \"ffac99be-3709-4c4f-9526-7c290027b026\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") "
Oct 30 03:27:58.783163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.783038    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-certs" pod "ffac99be-3709-4c4f-9526-7c290027b026" (UID: "ffac99be-3709-4c4f-9526-7c290027b026"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.783163 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.783058    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-credential-socket\") pod \"ffac99be-3709-4c4f-9526-7c290027b026\" (UID: \"ffac99be-3709-4c4f-9526-7c290027b026\") "
Oct 30 03:27:58.783643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.783615    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.783880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.783855    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-credential-socket" pod "ffac99be-3709-4c4f-9526-7c290027b026" (UID: "ffac99be-3709-4c4f-9526-7c290027b026"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.784002 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.783899    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffac99be-3709-4c4f-9526-7c290027b026-istiod-ca-cert" pod "ffac99be-3709-4c4f-9526-7c290027b026" (UID: "ffac99be-3709-4c4f-9526-7c290027b026"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:58.784077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.784016    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-data" pod "ffac99be-3709-4c4f-9526-7c290027b026" (UID: "ffac99be-3709-4c4f-9526-7c290027b026"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.784240 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.784213    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-socket" pod "ffac99be-3709-4c4f-9526-7c290027b026" (UID: "ffac99be-3709-4c4f-9526-7c290027b026"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.785405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.785373    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-kube-api-access-92zgz" pod "ffac99be-3709-4c4f-9526-7c290027b026" (UID: "ffac99be-3709-4c4f-9526-7c290027b026"). InnerVolumeSpecName "kube-api-access-92zgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:58.785559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.785403    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffac99be-3709-4c4f-9526-7c290027b026-istio-podinfo" pod "ffac99be-3709-4c4f-9526-7c290027b026" (UID: "ffac99be-3709-4c4f-9526-7c290027b026"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:27:58.785559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.785456    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-istio-token" pod "ffac99be-3709-4c4f-9526-7c290027b026" (UID: "ffac99be-3709-4c4f-9526-7c290027b026"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:58.785880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.785857    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-envoy" pod "ffac99be-3709-4c4f-9526-7c290027b026" (UID: "ffac99be-3709-4c4f-9526-7c290027b026"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.884605 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.884568    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffac99be-3709-4c4f-9526-7c290027b026-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.884848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.884599    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.884848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.884664    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.884848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.884674    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.884848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.884700    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffac99be-3709-4c4f-9526-7c290027b026-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.884848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.884709    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffac99be-3709-4c4f-9526-7c290027b026-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.884848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.884717    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.884848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:58.884727    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92zgz\" (UniqueName: \"kubernetes.io/projected/ffac99be-3709-4c4f-9526-7c290027b026-kube-api-access-92zgz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:27:59.054591 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.054548    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"]
Oct 30 03:27:59.069789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.069742    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"]
Oct 30 03:27:59.236800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.236765    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q" event={"ID":"95c27bd7-2869-4e56-9b87-a64522f6284e","Type":"ContainerStarted","Data":"3b9663598b1a839a32b9574fca1f87b1535e9ccac6758d74ca75a9520ca4db37"}
Oct 30 03:27:59.237016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.236823    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q" event={"ID":"95c27bd7-2869-4e56-9b87-a64522f6284e","Type":"ContainerStarted","Data":"e44d735adca9b548cad32b9177f4d84693d4223cecd6affa406e6241940ab515"}
Oct 30 03:27:59.238661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.238632    4278 generic.go:356] "Generic (PLEG): container finished" podID="ffac99be-3709-4c4f-9526-7c290027b026" containerID="86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60" exitCode=0
Oct 30 03:27:59.238802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.238697    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"
Oct 30 03:27:59.238802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.238699    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" event={"ID":"ffac99be-3709-4c4f-9526-7c290027b026","Type":"ContainerDied","Data":"86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60"}
Oct 30 03:27:59.238802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.238799    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr" event={"ID":"ffac99be-3709-4c4f-9526-7c290027b026","Type":"ContainerDied","Data":"3c5184c7c93490b8b201b812bf0de5c106c8dea0b8190160731e36560f44c0ca"}
Oct 30 03:27:59.238931 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.238823    4278 scope.go:122] "RemoveContainer" containerID="86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60"
Oct 30 03:27:59.250219 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.250201    4278 scope.go:122] "RemoveContainer" containerID="17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b"
Oct 30 03:27:59.261532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.261507    4278 scope.go:122] "RemoveContainer" containerID="86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60"
Oct 30 03:27:59.261809 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:27:59.261789    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60\": container with ID starting with 86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60 not found: ID does not exist" containerID="86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60"
Oct 30 03:27:59.261898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.261824    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60"} err="failed to get container status \"86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60\": rpc error: code = NotFound desc = could not find container \"86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60\": container with ID starting with 86d96f1c6a227d59992b2540df0f1b9ed80b31fad6243ed19d85500ad49dae60 not found: ID does not exist"
Oct 30 03:27:59.261898 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.261852    4278 scope.go:122] "RemoveContainer" containerID="17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b"
Oct 30 03:27:59.262110 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:27:59.262089    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b\": container with ID starting with 17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b not found: ID does not exist" containerID="17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b"
Oct 30 03:27:59.262151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.262117    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b"} err="failed to get container status \"17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b\": rpc error: code = NotFound desc = could not find container \"17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b\": container with ID starting with 17d92d2431d9eca939925367404853404c22439e229a0792822bf77e813f4c0b not found: ID does not exist"
Oct 30 03:27:59.278807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.278764    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q" podStartSLOduration=2.278751098 podStartE2EDuration="2.278751098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:59.277628824 +0000 UTC m=+2817.201837013" watchObservedRunningTime="2026-10-30 03:27:59.278751098 +0000 UTC m=+2817.202959286"
Oct 30 03:27:59.317165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.317083    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"]
Oct 30 03:27:59.324412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:27:59.324378    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-2l5dr"]
Oct 30 03:28:00.199848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:00.199817    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffac99be-3709-4c4f-9526-7c290027b026" path="/var/lib/kubelet/pods/ffac99be-3709-4c4f-9526-7c290027b026/volumes"
Oct 30 03:28:00.247490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:00.247450    4278 generic.go:356] "Generic (PLEG): container finished" podID="95c27bd7-2869-4e56-9b87-a64522f6284e" containerID="3b9663598b1a839a32b9574fca1f87b1535e9ccac6758d74ca75a9520ca4db37" exitCode=0
Oct 30 03:28:00.247716 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:00.247599    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q" event={"ID":"95c27bd7-2869-4e56-9b87-a64522f6284e","Type":"ContainerDied","Data":"3b9663598b1a839a32b9574fca1f87b1535e9ccac6758d74ca75a9520ca4db37"}
Oct 30 03:28:01.430829 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.430796    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:28:01.545303 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.544729    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-xks-ca\") pod \"95c27bd7-2869-4e56-9b87-a64522f6284e\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") "
Oct 30 03:28:01.545646 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.545616    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c27bd7-2869-4e56-9b87-a64522f6284e-kube-api-access-btpfs\" (UniqueName: \"kubernetes.io/projected/95c27bd7-2869-4e56-9b87-a64522f6284e-kube-api-access-btpfs\") pod \"95c27bd7-2869-4e56-9b87-a64522f6284e\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") "
Oct 30 03:28:01.545985 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.545945    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-openshift-ca\") pod \"95c27bd7-2869-4e56-9b87-a64522f6284e\" (UID: \"95c27bd7-2869-4e56-9b87-a64522f6284e\") "
Oct 30 03:28:01.546389 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.546361    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-openshift-ca" pod "95c27bd7-2869-4e56-9b87-a64522f6284e" (UID: "95c27bd7-2869-4e56-9b87-a64522f6284e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:01.547035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.546996    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:01.554646 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.552556    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-xks-ca" pod "95c27bd7-2869-4e56-9b87-a64522f6284e" (UID: "95c27bd7-2869-4e56-9b87-a64522f6284e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:01.554646 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.552684    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c27bd7-2869-4e56-9b87-a64522f6284e-kube-api-access-btpfs" pod "95c27bd7-2869-4e56-9b87-a64522f6284e" (UID: "95c27bd7-2869-4e56-9b87-a64522f6284e"). InnerVolumeSpecName "kube-api-access-btpfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:01.652451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.651453    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c27bd7-2869-4e56-9b87-a64522f6284e-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:01.652451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.651499    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpfs\" (UniqueName: \"kubernetes.io/projected/95c27bd7-2869-4e56-9b87-a64522f6284e-kube-api-access-btpfs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:01.891851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.891742    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"]
Oct 30 03:28:01.895150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:01.895112    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d9f3ac-n26bc"]
Oct 30 03:28:02.199379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:02.199296    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b796c3-f76b-4baf-8d8c-1431ab0b798e" path="/var/lib/kubelet/pods/02b796c3-f76b-4baf-8d8c-1431ab0b798e/volumes"
Oct 30 03:28:02.266808 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:02.266769    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q" event={"ID":"95c27bd7-2869-4e56-9b87-a64522f6284e","Type":"ContainerDied","Data":"e44d735adca9b548cad32b9177f4d84693d4223cecd6affa406e6241940ab515"}
Oct 30 03:28:02.266808 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:02.266803    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"
Oct 30 03:28:02.267096 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:02.266823    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e44d735adca9b548cad32b9177f4d84693d4223cecd6affa406e6241940ab515"
Oct 30 03:28:05.027010 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:05.026975    4278 scope.go:122] "RemoveContainer" containerID="a294afcba82899fc0535e97b81f440708e46a8a3c7002d6c24e0a442845fc75e"
Oct 30 03:28:08.830969 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:08.830930    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:28:09.253507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:09.252732    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"]
Oct 30 03:28:09.253507 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:09.252997    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy" containerID="cri-o://07e9289bf03a1c897ae879712a476f1122e809826c7dc24fe212ba67ffc1b2e2" gracePeriod=30
Oct 30 03:28:14.086490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.086411    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" podUID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" containerName="main" containerID="cri-o://820becf7904a6452ec59e8a953ad37706befcfda59b68a68c771deb8440ba193" gracePeriod=45
Oct 30 03:28:14.091169 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.090899    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main" containerID="cri-o://82d53b634f8ea244947919dd80c34587a7697e70a85a2442f6563b053112d62a" gracePeriod=45
Oct 30 03:28:14.411214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.411132    4278 generic.go:356] "Generic (PLEG): container finished" podID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerID="07e9289bf03a1c897ae879712a476f1122e809826c7dc24fe212ba67ffc1b2e2" exitCode=0
Oct 30 03:28:14.411451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.411310    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" event={"ID":"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc","Type":"ContainerDied","Data":"07e9289bf03a1c897ae879712a476f1122e809826c7dc24fe212ba67ffc1b2e2"}
Oct 30 03:28:14.414167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.414137    4278 generic.go:356] "Generic (PLEG): container finished" podID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerID="82d53b634f8ea244947919dd80c34587a7697e70a85a2442f6563b053112d62a" exitCode=0
Oct 30 03:28:14.414312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.414222    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" event={"ID":"48001cf3-8d2a-4d5f-b8f4-b341270c2100","Type":"ContainerDied","Data":"82d53b634f8ea244947919dd80c34587a7697e70a85a2442f6563b053112d62a"}
Oct 30 03:28:14.416350 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.416190    4278 generic.go:356] "Generic (PLEG): container finished" podID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" containerID="820becf7904a6452ec59e8a953ad37706befcfda59b68a68c771deb8440ba193" exitCode=0
Oct 30 03:28:14.416350 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.416317    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" event={"ID":"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327","Type":"ContainerDied","Data":"820becf7904a6452ec59e8a953ad37706befcfda59b68a68c771deb8440ba193"}
Oct 30 03:28:14.473978 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.473741    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:28:14.485052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.480517    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 03:28:14.551966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.551924    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tmp-dir\") pod \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") "
Oct 30 03:28:14.551966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.551962    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-home\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-home\") pod \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") "
Oct 30 03:28:14.552226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552030    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-dshm\") pod \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") "
Oct 30 03:28:14.552226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552072    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48001cf3-8d2a-4d5f-b8f4-b341270c2100-kube-api-access-nf9hh\" (UniqueName: \"kubernetes.io/projected/48001cf3-8d2a-4d5f-b8f4-b341270c2100-kube-api-access-nf9hh\") pod \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") "
Oct 30 03:28:14.552226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552123    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-home\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-home\") pod \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") "
Oct 30 03:28:14.552226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552163    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tmp-dir\") pod \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") "
Oct 30 03:28:14.552226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552200    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tls-certs\") pod \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") "
Oct 30 03:28:14.552579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552256    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-model-cache\") pod \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") "
Oct 30 03:28:14.552579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552254    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tmp-dir" pod "48001cf3-8d2a-4d5f-b8f4-b341270c2100" (UID: "48001cf3-8d2a-4d5f-b8f4-b341270c2100"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.552579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552284    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-dshm\") pod \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") "
Oct 30 03:28:14.552579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552328    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-home" pod "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" (UID: "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.552579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552348    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-kube-api-access-xnprh\" (UniqueName: \"kubernetes.io/projected/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-kube-api-access-xnprh\") pod \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") "
Oct 30 03:28:14.552579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552473    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tmp-dir" pod "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" (UID: "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.552579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552505    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-model-cache\") pod \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\" (UID: \"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327\") "
Oct 30 03:28:14.552579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552549    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tls-certs\" (UniqueName: \"kubernetes.io/secret/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tls-certs\") pod \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\" (UID: \"48001cf3-8d2a-4d5f-b8f4-b341270c2100\") "
Oct 30 03:28:14.553088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552610    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-home" pod "48001cf3-8d2a-4d5f-b8f4-b341270c2100" (UID: "48001cf3-8d2a-4d5f-b8f4-b341270c2100"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.553088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552687    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-model-cache" pod "48001cf3-8d2a-4d5f-b8f4-b341270c2100" (UID: "48001cf3-8d2a-4d5f-b8f4-b341270c2100"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.553088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.552842    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-model-cache" pod "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" (UID: "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.553088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.553049    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.553088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.553072    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.553088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.553086    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.553405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.553101    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.553405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.553112    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.553405 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.553124    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.555755 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.555705    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-dshm" pod "48001cf3-8d2a-4d5f-b8f4-b341270c2100" (UID: "48001cf3-8d2a-4d5f-b8f4-b341270c2100"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.557145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.557077    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-kube-api-access-xnprh" pod "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" (UID: "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327"). InnerVolumeSpecName "kube-api-access-xnprh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:14.558055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.558016    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48001cf3-8d2a-4d5f-b8f4-b341270c2100-kube-api-access-nf9hh" pod "48001cf3-8d2a-4d5f-b8f4-b341270c2100" (UID: "48001cf3-8d2a-4d5f-b8f4-b341270c2100"). InnerVolumeSpecName "kube-api-access-nf9hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:14.558715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.558689    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tls-certs" pod "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" (UID: "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:14.560946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.560906    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-dshm" pod "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" (UID: "c0a55e03-c9d8-4ef8-bdf3-3df6495cb327"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.561392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.561365    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tls-certs" pod "48001cf3-8d2a-4d5f-b8f4-b341270c2100" (UID: "48001cf3-8d2a-4d5f-b8f4-b341270c2100"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:14.622033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.621997    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 03:28:14.654143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654047    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-credential-socket\") pod \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") "
Oct 30 03:28:14.654143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654114    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istiod-ca-cert\") pod \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") "
Oct 30 03:28:14.654143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654141    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-envoy\") pod \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") "
Oct 30 03:28:14.654488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654318    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-podinfo\") pod \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") "
Oct 30 03:28:14.654488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654440    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-kube-api-access-h9hkf\" (UniqueName: \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-kube-api-access-h9hkf\") pod \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") "
Oct 30 03:28:14.654488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654447    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-credential-socket" pod "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" (UID: "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.654664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654556    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-token\" (UniqueName: \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-token\") pod \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") "
Oct 30 03:28:14.654664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654580    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istiod-ca-cert" pod "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" (UID: "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:14.654664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654589    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-certs\") pod \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") "
Oct 30 03:28:14.654664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654654    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-data\") pod \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") "
Oct 30 03:28:14.654859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654825    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-certs" pod "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" (UID: "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.654944 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654917    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-socket\") pod \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\" (UID: \"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc\") "
Oct 30 03:28:14.655018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.654970    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-data" pod "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" (UID: "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.655290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655267    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-socket" pod "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" (UID: "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.655372 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655354    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48001cf3-8d2a-4d5f-b8f4-b341270c2100-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.655454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655381    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9hh\" (UniqueName: \"kubernetes.io/projected/48001cf3-8d2a-4d5f-b8f4-b341270c2100-kube-api-access-nf9hh\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.655454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655420    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.655454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655450    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.655616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655465    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.655616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655479    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnprh\" (UniqueName: \"kubernetes.io/projected/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327-kube-api-access-xnprh\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.655616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655493    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.655616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655506    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48001cf3-8d2a-4d5f-b8f4-b341270c2100-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.655616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655525    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.655616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.655538    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.657333 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.657301    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-token" pod "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" (UID: "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:14.657516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.657396    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-envoy" pod "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" (UID: "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.657708 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.657683    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-podinfo" pod "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" (UID: "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:28:14.658032 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.658009    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-kube-api-access-h9hkf" pod "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" (UID: "ecb58a83-97bd-4072-9fb4-8a9754fa3fdc"). InnerVolumeSpecName "kube-api-access-h9hkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:14.756368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.756286    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.756368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.756316    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.756368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.756329    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.756368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.756337    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9hkf\" (UniqueName: \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-kube-api-access-h9hkf\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.756368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:14.756347    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:15.408584 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.408495    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" podUID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:28:15.424627 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.424596    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"
Oct 30 03:28:15.424797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.424594    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb" event={"ID":"ecb58a83-97bd-4072-9fb4-8a9754fa3fdc","Type":"ContainerDied","Data":"e62e724865765678b94f0e2e61ad1b3bba36185c5b684fb7876076c472e90d69"}
Oct 30 03:28:15.424797 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.424788    4278 scope.go:122] "RemoveContainer" containerID="07e9289bf03a1c897ae879712a476f1122e809826c7dc24fe212ba67ffc1b2e2"
Oct 30 03:28:15.426310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.426284    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd" event={"ID":"48001cf3-8d2a-4d5f-b8f4-b341270c2100","Type":"ContainerDied","Data":"987144823b38ac2fb26173f6c80edfa43338b968137db4e60f1b59f6177ee6c6"}
Oct 30 03:28:15.426397 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.426291    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"
Oct 30 03:28:15.428151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.428129    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"
Oct 30 03:28:15.428288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.428155    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54" event={"ID":"c0a55e03-c9d8-4ef8-bdf3-3df6495cb327","Type":"ContainerDied","Data":"795c51676f5e1f97d7e339b77e6ae871406f3aeb11c4cb2804d303f477dca167"}
Oct 30 03:28:15.437715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.437629    4278 scope.go:122] "RemoveContainer" containerID="82d53b634f8ea244947919dd80c34587a7697e70a85a2442f6563b053112d62a"
Oct 30 03:28:15.448054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.448033    4278 scope.go:122] "RemoveContainer" containerID="820becf7904a6452ec59e8a953ad37706befcfda59b68a68c771deb8440ba193"
Oct 30 03:28:15.486331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.486299    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"]
Oct 30 03:28:15.489645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.489615    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-w1-e11f03-kserve-5fbbd4f78d-skr54"]
Oct 30 03:28:15.521655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.521616    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"]
Oct 30 03:28:15.526622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.526588    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-ph4xb"]
Oct 30 03:28:15.558566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.558516    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"]
Oct 30 03:28:15.561728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:15.561698    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/premium-w1-e11f03-kserve-659dcc74b5-v6crd"]
Oct 30 03:28:16.199774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:16.199743    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" path="/var/lib/kubelet/pods/48001cf3-8d2a-4d5f-b8f4-b341270c2100/volumes"
Oct 30 03:28:16.200233 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:16.200220    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" path="/var/lib/kubelet/pods/c0a55e03-c9d8-4ef8-bdf3-3df6495cb327/volumes"
Oct 30 03:28:16.200683 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:16.200666    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" path="/var/lib/kubelet/pods/ecb58a83-97bd-4072-9fb4-8a9754fa3fdc/volumes"
Oct 30 03:28:24.191996 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:28:24.191952    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:28:26.987510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.987466    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"]
Oct 30 03:28:26.988038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988022    4278 cpu_manager.go:414] "containerMap: removing container" podUID="95c27bd7-2869-4e56-9b87-a64522f6284e" containerName="revoke-keys"
Oct 30 03:28:26.988084 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988039    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c27bd7-2869-4e56-9b87-a64522f6284e" containerName="revoke-keys"
Oct 30 03:28:26.988084 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988054    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" containerName="main"
Oct 30 03:28:26.988084 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988059    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" containerName="main"
Oct 30 03:28:26.988084 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988079    4278 cpu_manager.go:414] "containerMap: removing container" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main"
Oct 30 03:28:26.988216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988084    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main"
Oct 30 03:28:26.988216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988097    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy"
Oct 30 03:28:26.988216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988102    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy"
Oct 30 03:28:26.988216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988110    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy"
Oct 30 03:28:26.988216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988114    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy"
Oct 30 03:28:26.988216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988127    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy"
Oct 30 03:28:26.988216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988131    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy"
Oct 30 03:28:26.988216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988210    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c27bd7-2869-4e56-9b87-a64522f6284e" containerName="revoke-keys"
Oct 30 03:28:26.988216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988219    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a55e03-c9d8-4ef8-bdf3-3df6495cb327" containerName="main"
Oct 30 03:28:26.988523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988229    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="48001cf3-8d2a-4d5f-b8f4-b341270c2100" containerName="main"
Oct 30 03:28:26.988523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988237    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb58a83-97bd-4072-9fb4-8a9754fa3fdc" containerName="istio-proxy"
Oct 30 03:28:26.988523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988243    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy"
Oct 30 03:28:26.988523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.988249    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffac99be-3709-4c4f-9526-7c290027b026" containerName="istio-proxy"
Oct 30 03:28:26.993140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:26.993108    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.008526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.008496    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"]
Oct 30 03:28:27.012812 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.012779    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"]
Oct 30 03:28:27.012989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.012917    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.016475 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.016447    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:28:27.034378 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.034340    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"]
Oct 30 03:28:27.046616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.046578    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:28:27.130880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.130837    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cc055bd-nnb4f\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.131086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.130899    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjgp\" (UniqueName: \"kubernetes.io/projected/076479d2-e37d-48e8-8f4b-3a681fd5926a-kube-api-access-5zjgp\") pod \"maas-api-revoke-sub-5003bd41-rvtmg\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.131086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.131014    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5003bd41-rvtmg\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.131168 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.131128    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5003bd41-rvtmg\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.131221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.131203    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28ff7\" (UniqueName: \"kubernetes.io/projected/66960389-d77e-4b24-bf6e-2033ba1f7d21-kube-api-access-28ff7\") pod \"maas-api-revoke-sub-7cc055bd-nnb4f\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.131349 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.131324    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cc055bd-nnb4f\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.233054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.233010    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28ff7\" (UniqueName: \"kubernetes.io/projected/66960389-d77e-4b24-bf6e-2033ba1f7d21-kube-api-access-28ff7\") pod \"maas-api-revoke-sub-7cc055bd-nnb4f\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.233268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.233068    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cc055bd-nnb4f\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.233268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.233173    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cc055bd-nnb4f\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.233268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.233223    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjgp\" (UniqueName: \"kubernetes.io/projected/076479d2-e37d-48e8-8f4b-3a681fd5926a-kube-api-access-5zjgp\") pod \"maas-api-revoke-sub-5003bd41-rvtmg\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.233268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.233255    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5003bd41-rvtmg\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.233557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.233328    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5003bd41-rvtmg\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.234023 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.233990    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cc055bd-nnb4f\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.234151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.234128    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5003bd41-rvtmg\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.235941 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.235904    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5003bd41-rvtmg\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.236059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.235919    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cc055bd-nnb4f\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.252230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.252157    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjgp\" (UniqueName: \"kubernetes.io/projected/076479d2-e37d-48e8-8f4b-3a681fd5926a-kube-api-access-5zjgp\") pod \"maas-api-revoke-sub-5003bd41-rvtmg\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.252230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.252179    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28ff7\" (UniqueName: \"kubernetes.io/projected/66960389-d77e-4b24-bf6e-2033ba1f7d21-kube-api-access-28ff7\") pod \"maas-api-revoke-sub-7cc055bd-nnb4f\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.309638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.309203    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:27.335470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.334347    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:27.634834 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.634802    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"]
Oct 30 03:28:27.638911 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:28:27.638875    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076479d2_e37d_48e8_8f4b_3a681fd5926a.slice/crio-33039897c8e9dc2ac0fcdb52a49f3563f060f8c8bd4ff93d5bdc1320ce992662 WatchSource:0}: Error finding container 33039897c8e9dc2ac0fcdb52a49f3563f060f8c8bd4ff93d5bdc1320ce992662: Status 404 returned error can't find the container with id 33039897c8e9dc2ac0fcdb52a49f3563f060f8c8bd4ff93d5bdc1320ce992662
Oct 30 03:28:27.686208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.686165    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"]
Oct 30 03:28:27.692901 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:28:27.692872    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66960389_d77e_4b24_bf6e_2033ba1f7d21.slice/crio-a9d7ec219894d5e18157968e1324bd9056269cbec154995e88243fb1e2d2e7cc WatchSource:0}: Error finding container a9d7ec219894d5e18157968e1324bd9056269cbec154995e88243fb1e2d2e7cc: Status 404 returned error can't find the container with id a9d7ec219894d5e18157968e1324bd9056269cbec154995e88243fb1e2d2e7cc
Oct 30 03:28:27.736329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.736302    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"]
Oct 30 03:28:27.739652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:27.739624    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-4b5459-8w94q"]
Oct 30 03:28:28.198807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:28.198773    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c27bd7-2869-4e56-9b87-a64522f6284e" path="/var/lib/kubelet/pods/95c27bd7-2869-4e56-9b87-a64522f6284e/volumes"
Oct 30 03:28:28.535217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:28.535174    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f" event={"ID":"66960389-d77e-4b24-bf6e-2033ba1f7d21","Type":"ContainerStarted","Data":"5730a665967b9550c2200cb2d8ad6067f9c6d2b9241d8198eabf35f63e9206ed"}
Oct 30 03:28:28.535441 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:28.535255    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f" event={"ID":"66960389-d77e-4b24-bf6e-2033ba1f7d21","Type":"ContainerStarted","Data":"a9d7ec219894d5e18157968e1324bd9056269cbec154995e88243fb1e2d2e7cc"}
Oct 30 03:28:28.537026 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:28.537002    4278 generic.go:356] "Generic (PLEG): container finished" podID="076479d2-e37d-48e8-8f4b-3a681fd5926a" containerID="1059dd021e9c8f41f50ad6f03e7790c5e6abbbfc291d1ed321089b15376af1db" exitCode=0
Oct 30 03:28:28.537147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:28.537080    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg" event={"ID":"076479d2-e37d-48e8-8f4b-3a681fd5926a","Type":"ContainerDied","Data":"1059dd021e9c8f41f50ad6f03e7790c5e6abbbfc291d1ed321089b15376af1db"}
Oct 30 03:28:28.537147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:28.537120    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg" event={"ID":"076479d2-e37d-48e8-8f4b-3a681fd5926a","Type":"ContainerStarted","Data":"33039897c8e9dc2ac0fcdb52a49f3563f060f8c8bd4ff93d5bdc1320ce992662"}
Oct 30 03:28:28.638565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:28.638510    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f" podStartSLOduration=2.638493708 podStartE2EDuration="2.638493708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:28.592971347 +0000 UTC m=+2846.517179531" watchObservedRunningTime="2026-10-30 03:28:28.638493708 +0000 UTC m=+2846.562701932"
Oct 30 03:28:29.547697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.547656    4278 generic.go:356] "Generic (PLEG): container finished" podID="66960389-d77e-4b24-bf6e-2033ba1f7d21" containerID="5730a665967b9550c2200cb2d8ad6067f9c6d2b9241d8198eabf35f63e9206ed" exitCode=0
Oct 30 03:28:29.548170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.547731    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f" event={"ID":"66960389-d77e-4b24-bf6e-2033ba1f7d21","Type":"ContainerDied","Data":"5730a665967b9550c2200cb2d8ad6067f9c6d2b9241d8198eabf35f63e9206ed"}
Oct 30 03:28:29.744270 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.744241    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:29.765121 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.765084    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-openshift-ca\") pod \"076479d2-e37d-48e8-8f4b-3a681fd5926a\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") "
Oct 30 03:28:29.765306 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.765158    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-xks-ca\") pod \"076479d2-e37d-48e8-8f4b-3a681fd5926a\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") "
Oct 30 03:28:29.765306 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.765236    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076479d2-e37d-48e8-8f4b-3a681fd5926a-kube-api-access-5zjgp\" (UniqueName: \"kubernetes.io/projected/076479d2-e37d-48e8-8f4b-3a681fd5926a-kube-api-access-5zjgp\") pod \"076479d2-e37d-48e8-8f4b-3a681fd5926a\" (UID: \"076479d2-e37d-48e8-8f4b-3a681fd5926a\") "
Oct 30 03:28:29.765508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.765484    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-openshift-ca" pod "076479d2-e37d-48e8-8f4b-3a681fd5926a" (UID: "076479d2-e37d-48e8-8f4b-3a681fd5926a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:29.765622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.765605    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.767382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.767357    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-xks-ca" pod "076479d2-e37d-48e8-8f4b-3a681fd5926a" (UID: "076479d2-e37d-48e8-8f4b-3a681fd5926a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:29.767515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.767404    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076479d2-e37d-48e8-8f4b-3a681fd5926a-kube-api-access-5zjgp" pod "076479d2-e37d-48e8-8f4b-3a681fd5926a" (UID: "076479d2-e37d-48e8-8f4b-3a681fd5926a"). InnerVolumeSpecName "kube-api-access-5zjgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:29.866411 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.866318    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zjgp\" (UniqueName: \"kubernetes.io/projected/076479d2-e37d-48e8-8f4b-3a681fd5926a-kube-api-access-5zjgp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.866411 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:29.866350    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/076479d2-e37d-48e8-8f4b-3a681fd5926a-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.556242 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.556212    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"
Oct 30 03:28:30.556242 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.556231    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg" event={"ID":"076479d2-e37d-48e8-8f4b-3a681fd5926a","Type":"ContainerDied","Data":"33039897c8e9dc2ac0fcdb52a49f3563f060f8c8bd4ff93d5bdc1320ce992662"}
Oct 30 03:28:30.556763 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.556306    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33039897c8e9dc2ac0fcdb52a49f3563f060f8c8bd4ff93d5bdc1320ce992662"
Oct 30 03:28:30.751589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.751561    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:30.807112 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.807016    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-openshift-ca\") pod \"66960389-d77e-4b24-bf6e-2033ba1f7d21\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") "
Oct 30 03:28:30.807309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.807269    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66960389-d77e-4b24-bf6e-2033ba1f7d21-kube-api-access-28ff7\" (UniqueName: \"kubernetes.io/projected/66960389-d77e-4b24-bf6e-2033ba1f7d21-kube-api-access-28ff7\") pod \"66960389-d77e-4b24-bf6e-2033ba1f7d21\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") "
Oct 30 03:28:30.807399 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.807313    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-xks-ca\") pod \"66960389-d77e-4b24-bf6e-2033ba1f7d21\" (UID: \"66960389-d77e-4b24-bf6e-2033ba1f7d21\") "
Oct 30 03:28:30.808078 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.808048    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-openshift-ca" pod "66960389-d77e-4b24-bf6e-2033ba1f7d21" (UID: "66960389-d77e-4b24-bf6e-2033ba1f7d21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:30.811104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.811070    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-xks-ca" pod "66960389-d77e-4b24-bf6e-2033ba1f7d21" (UID: "66960389-d77e-4b24-bf6e-2033ba1f7d21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:30.811790 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.811753    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66960389-d77e-4b24-bf6e-2033ba1f7d21-kube-api-access-28ff7" pod "66960389-d77e-4b24-bf6e-2033ba1f7d21" (UID: "66960389-d77e-4b24-bf6e-2033ba1f7d21"). InnerVolumeSpecName "kube-api-access-28ff7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:30.908629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.908592    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28ff7\" (UniqueName: \"kubernetes.io/projected/66960389-d77e-4b24-bf6e-2033ba1f7d21-kube-api-access-28ff7\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.908629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.908627    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.908629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:30.908637    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66960389-d77e-4b24-bf6e-2033ba1f7d21-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:31.564804 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:31.564770    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"
Oct 30 03:28:31.564804 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:31.564792    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f" event={"ID":"66960389-d77e-4b24-bf6e-2033ba1f7d21","Type":"ContainerDied","Data":"a9d7ec219894d5e18157968e1324bd9056269cbec154995e88243fb1e2d2e7cc"}
Oct 30 03:28:31.565274 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:31.564840    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9d7ec219894d5e18157968e1324bd9056269cbec154995e88243fb1e2d2e7cc"
Oct 30 03:28:33.368715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.368678    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"]
Oct 30 03:28:33.369182 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.369168    4278 cpu_manager.go:414] "containerMap: removing container" podUID="66960389-d77e-4b24-bf6e-2033ba1f7d21" containerName="revoke-keys"
Oct 30 03:28:33.369228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.369182    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66960389-d77e-4b24-bf6e-2033ba1f7d21" containerName="revoke-keys"
Oct 30 03:28:33.369228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.369194    4278 cpu_manager.go:414] "containerMap: removing container" podUID="076479d2-e37d-48e8-8f4b-3a681fd5926a" containerName="revoke-keys"
Oct 30 03:28:33.369228 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.369201    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076479d2-e37d-48e8-8f4b-3a681fd5926a" containerName="revoke-keys"
Oct 30 03:28:33.369328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.369300    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="66960389-d77e-4b24-bf6e-2033ba1f7d21" containerName="revoke-keys"
Oct 30 03:28:33.369328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.369307    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="076479d2-e37d-48e8-8f4b-3a681fd5926a" containerName="revoke-keys"
Oct 30 03:28:33.372585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.372567    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.396488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.396458    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"]
Oct 30 03:28:33.403572 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.403542    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:28:33.403721 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.403673    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:28:33.439961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.439923    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czzzk\" (UniqueName: \"kubernetes.io/projected/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-kube-api-access-czzzk\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.440143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.439992    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.440143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.440130    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.541233 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.541194    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.541491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.541358    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czzzk\" (UniqueName: \"kubernetes.io/projected/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-kube-api-access-czzzk\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.541491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.541405    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.542117 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.542091    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.543769 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.543746    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.560053 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.560018    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czzzk\" (UniqueName: \"kubernetes.io/projected/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-kube-api-access-czzzk\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.686573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.686494    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:33.987205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:33.987166    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"]
Oct 30 03:28:33.990044 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:28:33.990001    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd764e10d_cadc_456e_b6d6_7d91d2d27ff5.slice/crio-e967d04e61dd35cff45afa6c52a206e508fbdf8140aa888ad57cd8e66958528a WatchSource:0}: Error finding container e967d04e61dd35cff45afa6c52a206e508fbdf8140aa888ad57cd8e66958528a: Status 404 returned error can't find the container with id e967d04e61dd35cff45afa6c52a206e508fbdf8140aa888ad57cd8e66958528a
Oct 30 03:28:34.589146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:34.589039    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh" event={"ID":"d764e10d-cadc-456e-b6d6-7d91d2d27ff5","Type":"ContainerStarted","Data":"4a07dbe76d6d8cb6588ac4c16973390eaa4135b8ee1523f7eee5f801e4a5a73b"}
Oct 30 03:28:34.589146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:34.589117    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh" event={"ID":"d764e10d-cadc-456e-b6d6-7d91d2d27ff5","Type":"ContainerStarted","Data":"e967d04e61dd35cff45afa6c52a206e508fbdf8140aa888ad57cd8e66958528a"}
Oct 30 03:28:34.628656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:34.628588    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh" podStartSLOduration=1.628571484 podStartE2EDuration="1.628571484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:34.627683552 +0000 UTC m=+2852.551891743" watchObservedRunningTime="2026-10-30 03:28:34.628571484 +0000 UTC m=+2852.552779672"
Oct 30 03:28:35.597274 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:35.597231    4278 generic.go:356] "Generic (PLEG): container finished" podID="d764e10d-cadc-456e-b6d6-7d91d2d27ff5" containerID="4a07dbe76d6d8cb6588ac4c16973390eaa4135b8ee1523f7eee5f801e4a5a73b" exitCode=0
Oct 30 03:28:35.597760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:35.597311    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh" event={"ID":"d764e10d-cadc-456e-b6d6-7d91d2d27ff5","Type":"ContainerDied","Data":"4a07dbe76d6d8cb6588ac4c16973390eaa4135b8ee1523f7eee5f801e4a5a73b"}
Oct 30 03:28:35.766870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:35.766827    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"]
Oct 30 03:28:35.767128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:35.767095    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy" containerID="cri-o://f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab" gracePeriod=30
Oct 30 03:28:36.222020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.221979    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"]
Oct 30 03:28:36.226793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.226769    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.262398 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.260677    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"]
Oct 30 03:28:36.324592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.319700    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.324592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.319846    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2l27\" (UniqueName: \"kubernetes.io/projected/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-kube-api-access-h2l27\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.324592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.319901    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.421292 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.421244    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2l27\" (UniqueName: \"kubernetes.io/projected/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-kube-api-access-h2l27\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.421561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.421330    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.421561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.421497    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.422471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.422407    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.432356 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.431982    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.453745 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.453709    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2l27\" (UniqueName: \"kubernetes.io/projected/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-kube-api-access-h2l27\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.549142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.548701    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:36.873636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.873220    4278 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:28:36.873636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.873284    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:28:36.883043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.880557    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:36.911492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:36.911238    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"]
Oct 30 03:28:36.914272 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:28:36.914235    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156ae6ed_95ff_43cf_ad7e_f8352e3829ff.slice/crio-3479de5f49382af5ff178d1ae1196d95bd0691918f3755d99563d725b59ca0e1 WatchSource:0}: Error finding container 3479de5f49382af5ff178d1ae1196d95bd0691918f3755d99563d725b59ca0e1: Status 404 returned error can't find the container with id 3479de5f49382af5ff178d1ae1196d95bd0691918f3755d99563d725b59ca0e1
Oct 30 03:28:37.028891 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.028850    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-xks-ca\") pod \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") "
Oct 30 03:28:37.029088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.028942    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-kube-api-access-czzzk\" (UniqueName: \"kubernetes.io/projected/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-kube-api-access-czzzk\") pod \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") "
Oct 30 03:28:37.029088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.029041    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-openshift-ca\") pod \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\" (UID: \"d764e10d-cadc-456e-b6d6-7d91d2d27ff5\") "
Oct 30 03:28:37.029443 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.029405    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-openshift-ca" pod "d764e10d-cadc-456e-b6d6-7d91d2d27ff5" (UID: "d764e10d-cadc-456e-b6d6-7d91d2d27ff5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:37.031788 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.031760    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-kube-api-access-czzzk" pod "d764e10d-cadc-456e-b6d6-7d91d2d27ff5" (UID: "d764e10d-cadc-456e-b6d6-7d91d2d27ff5"). InnerVolumeSpecName "kube-api-access-czzzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:37.031788 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.031764    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-xks-ca" pod "d764e10d-cadc-456e-b6d6-7d91d2d27ff5" (UID: "d764e10d-cadc-456e-b6d6-7d91d2d27ff5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:37.130234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.130135    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:37.130234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.130171    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czzzk\" (UniqueName: \"kubernetes.io/projected/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-kube-api-access-czzzk\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:37.130234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.130187    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d764e10d-cadc-456e-b6d6-7d91d2d27ff5-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:37.623569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.623522    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"
Oct 30 03:28:37.623749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.623558    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh" event={"ID":"d764e10d-cadc-456e-b6d6-7d91d2d27ff5","Type":"ContainerDied","Data":"e967d04e61dd35cff45afa6c52a206e508fbdf8140aa888ad57cd8e66958528a"}
Oct 30 03:28:37.623749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.623633    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e967d04e61dd35cff45afa6c52a206e508fbdf8140aa888ad57cd8e66958528a"
Oct 30 03:28:37.625391 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.625360    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb" event={"ID":"156ae6ed-95ff-43cf-ad7e-f8352e3829ff","Type":"ContainerStarted","Data":"d29a88c589e0745b371eed9b4dd54caee5ac146fbcf128dd63905b6b11ec32fb"}
Oct 30 03:28:37.625554 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.625401    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb" event={"ID":"156ae6ed-95ff-43cf-ad7e-f8352e3829ff","Type":"ContainerStarted","Data":"3479de5f49382af5ff178d1ae1196d95bd0691918f3755d99563d725b59ca0e1"}
Oct 30 03:28:37.664684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:37.664627    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb" podStartSLOduration=1.664610629 podStartE2EDuration="1.664610629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:37.663282648 +0000 UTC m=+2855.587490835" watchObservedRunningTime="2026-10-30 03:28:37.664610629 +0000 UTC m=+2855.588818817"
Oct 30 03:28:38.633671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:38.633631    4278 generic.go:356] "Generic (PLEG): container finished" podID="156ae6ed-95ff-43cf-ad7e-f8352e3829ff" containerID="d29a88c589e0745b371eed9b4dd54caee5ac146fbcf128dd63905b6b11ec32fb" exitCode=0
Oct 30 03:28:38.634130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:38.633726    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb" event={"ID":"156ae6ed-95ff-43cf-ad7e-f8352e3829ff","Type":"ContainerDied","Data":"d29a88c589e0745b371eed9b4dd54caee5ac146fbcf128dd63905b6b11ec32fb"}
Oct 30 03:28:39.785698 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:39.785671    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:39.965272 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:39.965154    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-xks-ca\") pod \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") "
Oct 30 03:28:39.965484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:39.965274    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-openshift-ca\") pod \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") "
Oct 30 03:28:39.965484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:39.965465    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-kube-api-access-h2l27\" (UniqueName: \"kubernetes.io/projected/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-kube-api-access-h2l27\") pod \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\" (UID: \"156ae6ed-95ff-43cf-ad7e-f8352e3829ff\") "
Oct 30 03:28:39.965741 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:39.965713    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-openshift-ca" pod "156ae6ed-95ff-43cf-ad7e-f8352e3829ff" (UID: "156ae6ed-95ff-43cf-ad7e-f8352e3829ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:39.965984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:39.965957    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:39.967648 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:39.967620    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-kube-api-access-h2l27" pod "156ae6ed-95ff-43cf-ad7e-f8352e3829ff" (UID: "156ae6ed-95ff-43cf-ad7e-f8352e3829ff"). InnerVolumeSpecName "kube-api-access-h2l27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:39.967736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:39.967673    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-xks-ca" pod "156ae6ed-95ff-43cf-ad7e-f8352e3829ff" (UID: "156ae6ed-95ff-43cf-ad7e-f8352e3829ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:40.066851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.066815    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2l27\" (UniqueName: \"kubernetes.io/projected/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-kube-api-access-h2l27\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:40.066851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.066849    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/156ae6ed-95ff-43cf-ad7e-f8352e3829ff-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:40.258563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.258526    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"]
Oct 30 03:28:40.259029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.259016    4278 cpu_manager.go:414] "containerMap: removing container" podUID="d764e10d-cadc-456e-b6d6-7d91d2d27ff5" containerName="revoke-keys"
Oct 30 03:28:40.259085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.259031    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d764e10d-cadc-456e-b6d6-7d91d2d27ff5" containerName="revoke-keys"
Oct 30 03:28:40.259085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.259049    4278 cpu_manager.go:414] "containerMap: removing container" podUID="156ae6ed-95ff-43cf-ad7e-f8352e3829ff" containerName="revoke-keys"
Oct 30 03:28:40.259085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.259054    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156ae6ed-95ff-43cf-ad7e-f8352e3829ff" containerName="revoke-keys"
Oct 30 03:28:40.259197 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.259141    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="d764e10d-cadc-456e-b6d6-7d91d2d27ff5" containerName="revoke-keys"
Oct 30 03:28:40.259197 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.259152    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="156ae6ed-95ff-43cf-ad7e-f8352e3829ff" containerName="revoke-keys"
Oct 30 03:28:40.262733 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.262715    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:28:40.281418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.281390    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3f72d5-dockercfg-m9g74\""
Oct 30 03:28:40.281642 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.281458    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3f72d5\""
Oct 30 03:28:40.291444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.285850    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"]
Oct 30 03:28:40.362016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.360630    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"]
Oct 30 03:28:40.366934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.366902    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:28:40.371774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.370563    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt\" (UID: \"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:28:40.388925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.373957    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8qv\" (UniqueName: \"kubernetes.io/projected/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-kube-api-access-cs8qv\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt\" (UID: \"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:28:40.393860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.393820    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"]
Oct 30 03:28:40.474981 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.474935    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8qv\" (UniqueName: \"kubernetes.io/projected/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-kube-api-access-cs8qv\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt\" (UID: \"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:28:40.475175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.475053    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt\" (UID: \"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:28:40.475175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.475130    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpx7\" (UniqueName: \"kubernetes.io/projected/20c80749-612c-4782-82fe-4746cc8b920c-kube-api-access-zgpx7\") pod \"payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf\" (UID: \"20c80749-612c-4782-82fe-4746cc8b920c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:28:40.475319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.475194    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c80749-612c-4782-82fe-4746cc8b920c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf\" (UID: \"20c80749-612c-4782-82fe-4746cc8b920c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:28:40.475851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.475826    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt\" (UID: \"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:28:40.494244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.494206    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8qv\" (UniqueName: \"kubernetes.io/projected/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-kube-api-access-cs8qv\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt\" (UID: \"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:28:40.577028 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.576938    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpx7\" (UniqueName: \"kubernetes.io/projected/20c80749-612c-4782-82fe-4746cc8b920c-kube-api-access-zgpx7\") pod \"payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf\" (UID: \"20c80749-612c-4782-82fe-4746cc8b920c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:28:40.577028 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.577001    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c80749-612c-4782-82fe-4746cc8b920c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf\" (UID: \"20c80749-612c-4782-82fe-4746cc8b920c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:28:40.577252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.577054    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:28:40.577766 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.577748    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c80749-612c-4782-82fe-4746cc8b920c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf\" (UID: \"20c80749-612c-4782-82fe-4746cc8b920c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:28:40.596382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.596358    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpx7\" (UniqueName: \"kubernetes.io/projected/20c80749-612c-4782-82fe-4746cc8b920c-kube-api-access-zgpx7\") pod \"payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf\" (UID: \"20c80749-612c-4782-82fe-4746cc8b920c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:28:40.651794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.651767    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"
Oct 30 03:28:40.651794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.651776    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb" event={"ID":"156ae6ed-95ff-43cf-ad7e-f8352e3829ff","Type":"ContainerDied","Data":"3479de5f49382af5ff178d1ae1196d95bd0691918f3755d99563d725b59ca0e1"}
Oct 30 03:28:40.652029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.651823    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3479de5f49382af5ff178d1ae1196d95bd0691918f3755d99563d725b59ca0e1"
Oct 30 03:28:40.703664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.703626    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:28:40.877096 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:40.877067    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"]
Oct 30 03:28:40.879535 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:28:40.879482    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386cc0d6_6b0d_4a45_972e_d2a8ba88e9a6.slice/crio-44ec85a61585b524909c35764d07ad9fe2bdb020b26fd5f438286529f5e6c25d WatchSource:0}: Error finding container 44ec85a61585b524909c35764d07ad9fe2bdb020b26fd5f438286529f5e6c25d: Status 404 returned error can't find the container with id 44ec85a61585b524909c35764d07ad9fe2bdb020b26fd5f438286529f5e6c25d
Oct 30 03:28:41.009781 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.009418    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"]
Oct 30 03:28:41.017807 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:28:41.017772    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c80749_612c_4782_82fe_4746cc8b920c.slice/crio-fccb48373836b139e6c5620f95725e2094d4f0935ba764c542b8dad6c0d898e9 WatchSource:0}: Error finding container fccb48373836b139e6c5620f95725e2094d4f0935ba764c542b8dad6c0d898e9: Status 404 returned error can't find the container with id fccb48373836b139e6c5620f95725e2094d4f0935ba764c542b8dad6c0d898e9
Oct 30 03:28:41.049756 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.049730    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:28:41.183149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183047    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-token\" (UniqueName: \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-token\") pod \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") "
Oct 30 03:28:41.183149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183104    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-kube-api-access-h929s\" (UniqueName: \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-kube-api-access-h929s\") pod \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") "
Oct 30 03:28:41.183149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183141    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-credential-socket\") pod \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") "
Oct 30 03:28:41.183515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183240    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istiod-ca-cert\") pod \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") "
Oct 30 03:28:41.183515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183315    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-socket\") pod \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") "
Oct 30 03:28:41.183515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183369    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-envoy\") pod \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") "
Oct 30 03:28:41.183515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183478    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-credential-socket" pod "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" (UID: "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:41.183515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183513    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-podinfo\") pod \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") "
Oct 30 03:28:41.183722 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183584    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-socket" pod "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" (UID: "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:41.183722 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183624    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-data\") pod \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") "
Oct 30 03:28:41.183722 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183666    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-certs\") pod \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\" (UID: \"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d\") "
Oct 30 03:28:41.183722 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183685    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istiod-ca-cert" pod "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" (UID: "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:41.183936 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.183892    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-data" pod "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" (UID: "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:41.184038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.184007    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-certs" pod "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" (UID: "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:41.184204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.184186    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.184283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.184210    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.184283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.184227    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.184283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.184241    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.184283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.184256    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.185611 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.185586    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-envoy" pod "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" (UID: "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:41.185611 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.185596    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-kube-api-access-h929s" pod "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" (UID: "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d"). InnerVolumeSpecName "kube-api-access-h929s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:41.185744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.185679    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-podinfo" pod "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" (UID: "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:28:41.185841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.185821    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-token" pod "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" (UID: "50a949e9-a9d1-480a-9cf7-d561bb0a2b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:41.285236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.285195    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.285236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.285229    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h929s\" (UniqueName: \"kubernetes.io/projected/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-kube-api-access-h929s\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.285236 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.285239    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.285561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.285250    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.662735 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.662697    4278 generic.go:356] "Generic (PLEG): container finished" podID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerID="f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab" exitCode=0
Oct 30 03:28:41.662964 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.662764    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"
Oct 30 03:28:41.662964 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.662784    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" event={"ID":"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d","Type":"ContainerDied","Data":"f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab"}
Oct 30 03:28:41.662964 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.662862    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6" event={"ID":"50a949e9-a9d1-480a-9cf7-d561bb0a2b6d","Type":"ContainerDied","Data":"bc549a9684ccc65832a018094a211adf4a7b00af7338708c1097098d5282c571"}
Oct 30 03:28:41.662964 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.662900    4278 scope.go:122] "RemoveContainer" containerID="f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab"
Oct 30 03:28:41.664813 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.664772    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt" event={"ID":"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6","Type":"ContainerStarted","Data":"c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9"}
Oct 30 03:28:41.664938 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.664837    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:28:41.664938 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.664862    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt" event={"ID":"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6","Type":"ContainerStarted","Data":"44ec85a61585b524909c35764d07ad9fe2bdb020b26fd5f438286529f5e6c25d"}
Oct 30 03:28:41.666939 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.666915    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf" event={"ID":"20c80749-612c-4782-82fe-4746cc8b920c","Type":"ContainerStarted","Data":"2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa"}
Oct 30 03:28:41.667082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.666961    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf" event={"ID":"20c80749-612c-4782-82fe-4746cc8b920c","Type":"ContainerStarted","Data":"fccb48373836b139e6c5620f95725e2094d4f0935ba764c542b8dad6c0d898e9"}
Oct 30 03:28:41.667082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.667028    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:28:41.674930 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.674912    4278 scope.go:122] "RemoveContainer" containerID="0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e"
Oct 30 03:28:41.685689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.685668    4278 scope.go:122] "RemoveContainer" containerID="f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab"
Oct 30 03:28:41.685987 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:28:41.685963    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab\": container with ID starting with f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab not found: ID does not exist" containerID="f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab"
Oct 30 03:28:41.686035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.685997    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab"} err="failed to get container status \"f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab\": rpc error: code = NotFound desc = could not find container \"f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab\": container with ID starting with f7d9372e347918a43f868719f6d12aa1fabb3e34489d3c761ce81c4c505630ab not found: ID does not exist"
Oct 30 03:28:41.686035 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.686016    4278 scope.go:122] "RemoveContainer" containerID="0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e"
Oct 30 03:28:41.686269 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:28:41.686249    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e\": container with ID starting with 0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e not found: ID does not exist" containerID="0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e"
Oct 30 03:28:41.686308 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.686276    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e"} err="failed to get container status \"0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e\": rpc error: code = NotFound desc = could not find container \"0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e\": container with ID starting with 0fc4df8d5f2992e1dfa6028dd550e7e786dc290d53ffb281c7c73cd292afe64e not found: ID does not exist"
Oct 30 03:28:41.716728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.716670    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt" podStartSLOduration=1.716655073 podStartE2EDuration="1.716655073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:41.713988199 +0000 UTC m=+2859.638196399" watchObservedRunningTime="2026-10-30 03:28:41.716655073 +0000 UTC m=+2859.640863260"
Oct 30 03:28:41.750568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.750501    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf" podStartSLOduration=1.750483732 podStartE2EDuration="1.750483732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:41.749924367 +0000 UTC m=+2859.674132557" watchObservedRunningTime="2026-10-30 03:28:41.750483732 +0000 UTC m=+2859.674691921"
Oct 30 03:28:41.787838 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.787807    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"]
Oct 30 03:28:41.791070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:41.791035    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-btjl6"]
Oct 30 03:28:42.199547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:42.199505    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" path="/var/lib/kubelet/pods/50a949e9-a9d1-480a-9cf7-d561bb0a2b6d/volumes"
Oct 30 03:28:43.011304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:43.011257    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:28:44.427705 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.427665    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"]
Oct 30 03:28:44.428229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.427894    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk" podUID="f07c99c1-1acb-46cd-be80-25c6ba04dae3" containerName="maas-api" containerID="cri-o://d4abcab6e68e4289d240281dcf2571e878465b12d46119d65bb5d3e1aeb85e57" gracePeriod=30
Oct 30 03:28:44.702635 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.702497    4278 generic.go:356] "Generic (PLEG): container finished" podID="f07c99c1-1acb-46cd-be80-25c6ba04dae3" containerID="d4abcab6e68e4289d240281dcf2571e878465b12d46119d65bb5d3e1aeb85e57" exitCode=0
Oct 30 03:28:44.702635 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.702589    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk" event={"ID":"f07c99c1-1acb-46cd-be80-25c6ba04dae3","Type":"ContainerDied","Data":"d4abcab6e68e4289d240281dcf2571e878465b12d46119d65bb5d3e1aeb85e57"}
Oct 30 03:28:44.733780 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.733755    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:28:44.829810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.829768    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-metrics-certs\") pod \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") "
Oct 30 03:28:44.830011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.829828    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07c99c1-1acb-46cd-be80-25c6ba04dae3-kube-api-access-pcbtr\" (UniqueName: \"kubernetes.io/projected/f07c99c1-1acb-46cd-be80-25c6ba04dae3-kube-api-access-pcbtr\") pod \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") "
Oct 30 03:28:44.830011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.829906    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07c99c1-1acb-46cd-be80-25c6ba04dae3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07c99c1-1acb-46cd-be80-25c6ba04dae3-openshift-service-ca\") pod \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") "
Oct 30 03:28:44.830011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.830008    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-maas-api-tls\") pod \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\" (UID: \"f07c99c1-1acb-46cd-be80-25c6ba04dae3\") "
Oct 30 03:28:44.830315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.830286    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07c99c1-1acb-46cd-be80-25c6ba04dae3-openshift-service-ca" pod "f07c99c1-1acb-46cd-be80-25c6ba04dae3" (UID: "f07c99c1-1acb-46cd-be80-25c6ba04dae3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:44.832208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.832178    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-metrics-certs" pod "f07c99c1-1acb-46cd-be80-25c6ba04dae3" (UID: "f07c99c1-1acb-46cd-be80-25c6ba04dae3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:44.832323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.832236    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-maas-api-tls" pod "f07c99c1-1acb-46cd-be80-25c6ba04dae3" (UID: "f07c99c1-1acb-46cd-be80-25c6ba04dae3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:44.832323 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.832255    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07c99c1-1acb-46cd-be80-25c6ba04dae3-kube-api-access-pcbtr" pod "f07c99c1-1acb-46cd-be80-25c6ba04dae3" (UID: "f07c99c1-1acb-46cd-be80-25c6ba04dae3"). InnerVolumeSpecName "kube-api-access-pcbtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:44.936518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.931520    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcbtr\" (UniqueName: \"kubernetes.io/projected/f07c99c1-1acb-46cd-be80-25c6ba04dae3-kube-api-access-pcbtr\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:44.936518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.931577    4278 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07c99c1-1acb-46cd-be80-25c6ba04dae3-openshift-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:44.936518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.931595    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-maas-api-tls\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:44.936518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:44.931610    4278 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f07c99c1-1acb-46cd-be80-25c6ba04dae3-metrics-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:45.273240 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:45.273201    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"]
Oct 30 03:28:45.283833 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:45.283796    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9528f-g2bdh"]
Oct 30 03:28:45.711680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:45.711594    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"
Oct 30 03:28:45.712181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:45.711584    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk" event={"ID":"f07c99c1-1acb-46cd-be80-25c6ba04dae3","Type":"ContainerDied","Data":"395816f7313e4d4005beccb951563593f67070281ef1a7a14837f724718639d1"}
Oct 30 03:28:45.712181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:45.711730    4278 scope.go:122] "RemoveContainer" containerID="d4abcab6e68e4289d240281dcf2571e878465b12d46119d65bb5d3e1aeb85e57"
Oct 30 03:28:45.764912 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:45.764874    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"]
Oct 30 03:28:45.775867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:45.775832    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9528f-66d7479966-v8bqk"]
Oct 30 03:28:46.198282 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:46.198250    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d764e10d-cadc-456e-b6d6-7d91d2d27ff5" path="/var/lib/kubelet/pods/d764e10d-cadc-456e-b6d6-7d91d2d27ff5/volumes"
Oct 30 03:28:46.198852 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:46.198830    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07c99c1-1acb-46cd-be80-25c6ba04dae3" path="/var/lib/kubelet/pods/f07c99c1-1acb-46cd-be80-25c6ba04dae3/volumes"
Oct 30 03:28:49.149514 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:49.149473    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"]
Oct 30 03:28:49.160094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:49.160060    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e11f03-7mm7p"]
Oct 30 03:28:50.200525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:50.200490    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288dc87e-bf37-47bc-9f9f-5e500e6c1308" path="/var/lib/kubelet/pods/288dc87e-bf37-47bc-9f9f-5e500e6c1308/volumes"
Oct 30 03:28:52.679504 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.679474    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:28:52.679951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.679557    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:28:52.948704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.945561    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"]
Oct 30 03:28:52.948704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.947391    4278 cpu_manager.go:414] "containerMap: removing container" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy"
Oct 30 03:28:52.948704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.947437    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy"
Oct 30 03:28:52.948704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.947455    4278 cpu_manager.go:414] "containerMap: removing container" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy"
Oct 30 03:28:52.948704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.947464    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy"
Oct 30 03:28:52.948704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.947508    4278 cpu_manager.go:414] "containerMap: removing container" podUID="f07c99c1-1acb-46cd-be80-25c6ba04dae3" containerName="maas-api"
Oct 30 03:28:52.948704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.947515    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07c99c1-1acb-46cd-be80-25c6ba04dae3" containerName="maas-api"
Oct 30 03:28:52.948704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.947682    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07c99c1-1acb-46cd-be80-25c6ba04dae3" containerName="maas-api"
Oct 30 03:28:52.948704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.947708    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy"
Oct 30 03:28:52.948704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.947718    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a949e9-a9d1-480a-9cf7-d561bb0a2b6d" containerName="istio-proxy"
Oct 30 03:28:52.956394 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.956350    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:52.974057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.969357    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"]
Oct 30 03:28:52.974057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.971160    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:28:52.974057 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.971298    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:28:52.997446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.997396    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:28:52.997627 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:52.997450    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:28:53.038195 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.038153    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:53.038393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.038218    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:53.038393 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.038338    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlm2\" (UniqueName: \"kubernetes.io/projected/4920cac7-e851-463e-a98d-3d765f31c0dc-kube-api-access-xxlm2\") pod \"maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:53.139798 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.139761    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:53.139991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.139846    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:53.139991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.139917    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlm2\" (UniqueName: \"kubernetes.io/projected/4920cac7-e851-463e-a98d-3d765f31c0dc-kube-api-access-xxlm2\") pod \"maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:53.140694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.140668    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:53.142727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.142697    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:53.158450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.158396    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlm2\" (UniqueName: \"kubernetes.io/projected/4920cac7-e851-463e-a98d-3d765f31c0dc-kube-api-access-xxlm2\") pod \"maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:53.281867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.281828    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:53.588614 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.588583    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"]
Oct 30 03:28:53.590833 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:28:53.590795    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4920cac7_e851_463e_a98d_3d765f31c0dc.slice/crio-cfad6d494debb29d9493f76504168e7cb9bfc5fdc43771d6fdfebcd20f6f8706 WatchSource:0}: Error finding container cfad6d494debb29d9493f76504168e7cb9bfc5fdc43771d6fdfebcd20f6f8706: Status 404 returned error can't find the container with id cfad6d494debb29d9493f76504168e7cb9bfc5fdc43771d6fdfebcd20f6f8706
Oct 30 03:28:53.774302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:53.774254    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn" event={"ID":"4920cac7-e851-463e-a98d-3d765f31c0dc","Type":"ContainerStarted","Data":"cfad6d494debb29d9493f76504168e7cb9bfc5fdc43771d6fdfebcd20f6f8706"}
Oct 30 03:28:54.190480 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:28:54.190378    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:28:54.827495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:54.818079    4278 generic.go:356] "Generic (PLEG): container finished" podID="4920cac7-e851-463e-a98d-3d765f31c0dc" containerID="dabb0a273a9ede8151397177edc2de6354805dd83dc50059f2556a7496f70ebe" exitCode=0
Oct 30 03:28:54.827495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:54.818297    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn" event={"ID":"4920cac7-e851-463e-a98d-3d765f31c0dc","Type":"ContainerDied","Data":"dabb0a273a9ede8151397177edc2de6354805dd83dc50059f2556a7496f70ebe"}
Oct 30 03:28:55.281210 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:55.281172    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"]
Oct 30 03:28:55.284444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:55.284405    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-8b0787-drhvb"]
Oct 30 03:28:55.993727 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:55.993699    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:56.076632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.076526    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-openshift-ca\") pod \"4920cac7-e851-463e-a98d-3d765f31c0dc\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") "
Oct 30 03:28:56.076632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.076615    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-xks-ca\") pod \"4920cac7-e851-463e-a98d-3d765f31c0dc\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") "
Oct 30 03:28:56.076897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.076721    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4920cac7-e851-463e-a98d-3d765f31c0dc-kube-api-access-xxlm2\" (UniqueName: \"kubernetes.io/projected/4920cac7-e851-463e-a98d-3d765f31c0dc-kube-api-access-xxlm2\") pod \"4920cac7-e851-463e-a98d-3d765f31c0dc\" (UID: \"4920cac7-e851-463e-a98d-3d765f31c0dc\") "
Oct 30 03:28:56.077015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.076982    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-openshift-ca" pod "4920cac7-e851-463e-a98d-3d765f31c0dc" (UID: "4920cac7-e851-463e-a98d-3d765f31c0dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:56.077313 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.077290    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:56.080477 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.080095    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-xks-ca" pod "4920cac7-e851-463e-a98d-3d765f31c0dc" (UID: "4920cac7-e851-463e-a98d-3d765f31c0dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:56.081772 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.081743    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4920cac7-e851-463e-a98d-3d765f31c0dc-kube-api-access-xxlm2" pod "4920cac7-e851-463e-a98d-3d765f31c0dc" (UID: "4920cac7-e851-463e-a98d-3d765f31c0dc"). InnerVolumeSpecName "kube-api-access-xxlm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:56.178835 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.178794    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4920cac7-e851-463e-a98d-3d765f31c0dc-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:56.178835 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.178829    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxlm2\" (UniqueName: \"kubernetes.io/projected/4920cac7-e851-463e-a98d-3d765f31c0dc-kube-api-access-xxlm2\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:28:56.198640 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.198602    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156ae6ed-95ff-43cf-ad7e-f8352e3829ff" path="/var/lib/kubelet/pods/156ae6ed-95ff-43cf-ad7e-f8352e3829ff/volumes"
Oct 30 03:28:56.833083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.833016    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"
Oct 30 03:28:56.833288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.833037    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn" event={"ID":"4920cac7-e851-463e-a98d-3d765f31c0dc","Type":"ContainerDied","Data":"cfad6d494debb29d9493f76504168e7cb9bfc5fdc43771d6fdfebcd20f6f8706"}
Oct 30 03:28:56.833288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:56.833186    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfad6d494debb29d9493f76504168e7cb9bfc5fdc43771d6fdfebcd20f6f8706"
Oct 30 03:28:57.026245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:57.026208    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"]
Oct 30 03:28:57.029352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:57.029317    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7413d497-2v4gt"]
Oct 30 03:28:58.199078 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:28:58.199041    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c797b9d-8a97-4d54-ab28-9da20105a44a" path="/var/lib/kubelet/pods/7c797b9d-8a97-4d54-ab28-9da20105a44a/volumes"
Oct 30 03:29:02.842549 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.842506    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"]
Oct 30 03:29:02.843033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.843019    4278 cpu_manager.go:414] "containerMap: removing container" podUID="4920cac7-e851-463e-a98d-3d765f31c0dc" containerName="revoke-keys"
Oct 30 03:29:02.843084 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.843034    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4920cac7-e851-463e-a98d-3d765f31c0dc" containerName="revoke-keys"
Oct 30 03:29:02.843155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.843145    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="4920cac7-e851-463e-a98d-3d765f31c0dc" containerName="revoke-keys"
Oct 30 03:29:02.848052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.848028    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:02.865597 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.865569    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:29:02.865783 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.865608    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:29:02.877596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.877565    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:29:02.877596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.877596    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:29:02.901050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.901016    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"]
Oct 30 03:29:02.964003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.963963    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:02.964231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.964197    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:02.964343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:02.964315    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8bp\" (UniqueName: \"kubernetes.io/projected/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-kube-api-access-5p8bp\") pod \"maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:03.065819 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.065774    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:03.066060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.065853    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8bp\" (UniqueName: \"kubernetes.io/projected/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-kube-api-access-5p8bp\") pod \"maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:03.066060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.065892    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:03.066619 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.066591    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:03.068391 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.068358    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:03.085663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.085633    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8bp\" (UniqueName: \"kubernetes.io/projected/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-kube-api-access-5p8bp\") pod \"maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:03.162314 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.162220    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:03.469971 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:29:03.469929    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d0ddf1_eafd_4cab_a877_d4d6848ca7bc.slice/crio-92ad7f79e75bad306ffc407be4b35b80238d554d62835ad08162d1c317a97916 WatchSource:0}: Error finding container 92ad7f79e75bad306ffc407be4b35b80238d554d62835ad08162d1c317a97916: Status 404 returned error can't find the container with id 92ad7f79e75bad306ffc407be4b35b80238d554d62835ad08162d1c317a97916
Oct 30 03:29:03.478344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.478307    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"]
Oct 30 03:29:03.887670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.887633    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj" event={"ID":"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc","Type":"ContainerStarted","Data":"33da365a13fda43484786b4a57c13b8eef92dea49f695239c329448631d5071e"}
Oct 30 03:29:03.888103 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.887693    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj" event={"ID":"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc","Type":"ContainerStarted","Data":"92ad7f79e75bad306ffc407be4b35b80238d554d62835ad08162d1c317a97916"}
Oct 30 03:29:03.928957 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:03.928895    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj" podStartSLOduration=1.928878299 podStartE2EDuration="1.928878299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:03.925288439 +0000 UTC m=+2881.849496627" watchObservedRunningTime="2026-10-30 03:29:03.928878299 +0000 UTC m=+2881.853086525"
Oct 30 03:29:04.896480 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:04.896444    4278 generic.go:356] "Generic (PLEG): container finished" podID="a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc" containerID="33da365a13fda43484786b4a57c13b8eef92dea49f695239c329448631d5071e" exitCode=0
Oct 30 03:29:04.896924 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:04.896519    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj" event={"ID":"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc","Type":"ContainerDied","Data":"33da365a13fda43484786b4a57c13b8eef92dea49f695239c329448631d5071e"}
Oct 30 03:29:05.088951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:05.088906    4278 scope.go:122] "RemoveContainer" containerID="7eb87eea662ef9f96a54e1cea75edb876b40b0cb16ee4f3d285ca10d4684f0e2"
Oct 30 03:29:05.099345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:05.099305    4278 scope.go:122] "RemoveContainer" containerID="8995c4d042ac94f449fdc49a211f3bdb18a07837110eba5fe299db12396e3f9d"
Oct 30 03:29:05.109295 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:05.109272    4278 scope.go:122] "RemoveContainer" containerID="a30052cb5784f695d5266e342b4acc800f952a301ba60bbf4736aed324d7c0bc"
Oct 30 03:29:06.043903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.043877    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:06.201761 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.201669    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-openshift-ca\") pod \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") "
Oct 30 03:29:06.201761 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.201740    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-xks-ca\") pod \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") "
Oct 30 03:29:06.202024 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.201777    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-kube-api-access-5p8bp\" (UniqueName: \"kubernetes.io/projected/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-kube-api-access-5p8bp\") pod \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\" (UID: \"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc\") "
Oct 30 03:29:06.202128 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.202098    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-openshift-ca" pod "a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc" (UID: "a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:06.203948 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.203912    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-xks-ca" pod "a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc" (UID: "a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:06.204108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.203972    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-kube-api-access-5p8bp" pod "a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc" (UID: "a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc"). InnerVolumeSpecName "kube-api-access-5p8bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:06.302961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.302910    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:06.302961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.302947    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:06.302961 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.302962    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p8bp\" (UniqueName: \"kubernetes.io/projected/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc-kube-api-access-5p8bp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:06.911677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.911641    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"
Oct 30 03:29:06.911677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.911666    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj" event={"ID":"a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc","Type":"ContainerDied","Data":"92ad7f79e75bad306ffc407be4b35b80238d554d62835ad08162d1c317a97916"}
Oct 30 03:29:06.911927 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:06.911721    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92ad7f79e75bad306ffc407be4b35b80238d554d62835ad08162d1c317a97916"
Oct 30 03:29:07.026278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:07.026237    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"]
Oct 30 03:29:07.029440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:07.029404    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33ffa757-sh7bn"]
Oct 30 03:29:07.574033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:07.573990    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"]
Oct 30 03:29:07.602272 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:07.602236    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"]
Oct 30 03:29:08.198411 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:08.198374    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c87bcc-494c-42cf-ace8-3c4692d53c47" path="/var/lib/kubelet/pods/f1c87bcc-494c-42cf-ace8-3c4692d53c47/volumes"
Oct 30 03:29:13.300655 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:13.300612    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:29:15.035153 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:15.035102    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"]
Oct 30 03:29:15.046251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:15.046218    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-055daa-kbkzn"]
Oct 30 03:29:16.198744 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:16.198705    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4920cac7-e851-463e-a98d-3d765f31c0dc" path="/var/lib/kubelet/pods/4920cac7-e851-463e-a98d-3d765f31c0dc/volumes"
Oct 30 03:29:19.946832 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:19.946793    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"]
Oct 30 03:29:20.011386 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:20.011349    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"]
Oct 30 03:29:22.189461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.189399    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"]
Oct 30 03:29:22.189984 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.189966    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc" containerName="revoke-keys"
Oct 30 03:29:22.190026 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.189985    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc" containerName="revoke-keys"
Oct 30 03:29:22.190089 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.190080    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc" containerName="revoke-keys"
Oct 30 03:29:22.195367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.195340    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.212952 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.212915    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"]
Oct 30 03:29:22.220516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.220474    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:29:22.220691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.220546    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:29:22.261645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.261608    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:29:22.261645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.261651    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:29:22.293715 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.293677    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.293945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.293754    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfs5d\" (UniqueName: \"kubernetes.io/projected/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-kube-api-access-rfs5d\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.293945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.293898    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.395120 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.395079    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.395353 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.395137    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfs5d\" (UniqueName: \"kubernetes.io/projected/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-kube-api-access-rfs5d\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.395353 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.395186    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.395965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.395911    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.397728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.397707    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.413774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.413741    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfs5d\" (UniqueName: \"kubernetes.io/projected/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-kube-api-access-rfs5d\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.510206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.510171    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:22.596991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.596950    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" containerID="cri-o://e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2" gracePeriod=45
Oct 30 03:29:22.626155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.625995    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main" containerID="cri-o://45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55" gracePeriod=45
Oct 30 03:29:22.657649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.657608    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": dial tcp 10.134.0.80:8000: connect: connection refused"
Oct 30 03:29:22.841893 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.841848    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"]
Oct 30 03:29:22.843457 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:29:22.842569    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bef6098_76d0_4f91_9ea8_8c80652f9ce6.slice/crio-9bd1a35f1572ede887bb670e3bc2a60d7f2e51159ac2a3fe70c93605510b9abd WatchSource:0}: Error finding container 9bd1a35f1572ede887bb670e3bc2a60d7f2e51159ac2a3fe70c93605510b9abd: Status 404 returned error can't find the container with id 9bd1a35f1572ede887bb670e3bc2a60d7f2e51159ac2a3fe70c93605510b9abd
Oct 30 03:29:22.906492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.906462    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 03:29:22.944778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:22.944748    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 03:29:23.005811 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.005764    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-kube-api-access-k4bsv\" (UniqueName: \"kubernetes.io/projected/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-kube-api-access-k4bsv\") pod \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") "
Oct 30 03:29:23.006020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.005866    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tmp-dir\") pod \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") "
Oct 30 03:29:23.006020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.005904    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-dshm\") pod \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") "
Oct 30 03:29:23.006226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.006193    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tmp-dir" pod "b4584e9e-386b-47a3-93e3-abe7a1961f4c" (UID: "b4584e9e-386b-47a3-93e3-abe7a1961f4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.006463 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.006394    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tmp-dir\") pod \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") "
Oct 30 03:29:23.006595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.006519    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tls-certs\") pod \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") "
Oct 30 03:29:23.006740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.006714    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tmp-dir" pod "b215a820-a1ac-49d0-9ad5-cd8a41b54d15" (UID: "b215a820-a1ac-49d0-9ad5-cd8a41b54d15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.006934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.006906    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-home\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-home\") pod \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") "
Oct 30 03:29:23.007024 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.006990    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tls-certs\" (UniqueName: \"kubernetes.io/secret/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tls-certs\") pod \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") "
Oct 30 03:29:23.007090 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.007068    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-home\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-home\") pod \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") "
Oct 30 03:29:23.007162 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.007115    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-model-cache\") pod \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") "
Oct 30 03:29:23.007229 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.007190    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4584e9e-386b-47a3-93e3-abe7a1961f4c-kube-api-access-q6jjp\" (UniqueName: \"kubernetes.io/projected/b4584e9e-386b-47a3-93e3-abe7a1961f4c-kube-api-access-q6jjp\") pod \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") "
Oct 30 03:29:23.007290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.007266    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-model-cache\") pod \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\" (UID: \"b215a820-a1ac-49d0-9ad5-cd8a41b54d15\") "
Oct 30 03:29:23.007351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.007342    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-dshm\") pod \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\" (UID: \"b4584e9e-386b-47a3-93e3-abe7a1961f4c\") "
Oct 30 03:29:23.007665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.007642    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-home" pod "b215a820-a1ac-49d0-9ad5-cd8a41b54d15" (UID: "b215a820-a1ac-49d0-9ad5-cd8a41b54d15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.008200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.007888    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-model-cache" pod "b4584e9e-386b-47a3-93e3-abe7a1961f4c" (UID: "b4584e9e-386b-47a3-93e3-abe7a1961f4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.008200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.008040    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.008200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.008065    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.008200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.008066    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-model-cache" pod "b215a820-a1ac-49d0-9ad5-cd8a41b54d15" (UID: "b215a820-a1ac-49d0-9ad5-cd8a41b54d15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.008200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.008080    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.008200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.008140    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.008200 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.008170    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-home" pod "b4584e9e-386b-47a3-93e3-abe7a1961f4c" (UID: "b4584e9e-386b-47a3-93e3-abe7a1961f4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.010137 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.010106    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tls-certs" pod "b4584e9e-386b-47a3-93e3-abe7a1961f4c" (UID: "b4584e9e-386b-47a3-93e3-abe7a1961f4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:23.010137 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.010120    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-kube-api-access-k4bsv" pod "b215a820-a1ac-49d0-9ad5-cd8a41b54d15" (UID: "b215a820-a1ac-49d0-9ad5-cd8a41b54d15"). InnerVolumeSpecName "kube-api-access-k4bsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:23.010316 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.010237    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4584e9e-386b-47a3-93e3-abe7a1961f4c-kube-api-access-q6jjp" pod "b4584e9e-386b-47a3-93e3-abe7a1961f4c" (UID: "b4584e9e-386b-47a3-93e3-abe7a1961f4c"). InnerVolumeSpecName "kube-api-access-q6jjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:23.010379 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.010311    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-dshm" pod "b215a820-a1ac-49d0-9ad5-cd8a41b54d15" (UID: "b215a820-a1ac-49d0-9ad5-cd8a41b54d15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.010589 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.010564    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-dshm" pod "b4584e9e-386b-47a3-93e3-abe7a1961f4c" (UID: "b4584e9e-386b-47a3-93e3-abe7a1961f4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.010762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.010741    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tls-certs" pod "b215a820-a1ac-49d0-9ad5-cd8a41b54d15" (UID: "b215a820-a1ac-49d0-9ad5-cd8a41b54d15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:23.021752 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.021719    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh" event={"ID":"8bef6098-76d0-4f91-9ea8-8c80652f9ce6","Type":"ContainerStarted","Data":"9bd1a35f1572ede887bb670e3bc2a60d7f2e51159ac2a3fe70c93605510b9abd"}
Oct 30 03:29:23.023354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.023326    4278 generic.go:356] "Generic (PLEG): container finished" podID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerID="45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55" exitCode=0
Oct 30 03:29:23.023519 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.023503    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" event={"ID":"b4584e9e-386b-47a3-93e3-abe7a1961f4c","Type":"ContainerDied","Data":"45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55"}
Oct 30 03:29:23.023590 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.023541    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz" event={"ID":"b4584e9e-386b-47a3-93e3-abe7a1961f4c","Type":"ContainerDied","Data":"768a01277591fe3ab337a184e78c234b9c96abc494da8ea4c1149da18bfd30e3"}
Oct 30 03:29:23.023590 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.023558    4278 scope.go:122] "RemoveContainer" containerID="45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55"
Oct 30 03:29:23.023685 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.023675    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"
Oct 30 03:29:23.031478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.027884    4278 generic.go:356] "Generic (PLEG): container finished" podID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerID="e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2" exitCode=0
Oct 30 03:29:23.031478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.027921    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"
Oct 30 03:29:23.031478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.027940    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" event={"ID":"b215a820-a1ac-49d0-9ad5-cd8a41b54d15","Type":"ContainerDied","Data":"e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2"}
Oct 30 03:29:23.031478 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.028002    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj" event={"ID":"b215a820-a1ac-49d0-9ad5-cd8a41b54d15","Type":"ContainerDied","Data":"7c8977e9b184c15f58a23e59ab53aff8ce0a4b904148e2403a20bef6b0a1b1dc"}
Oct 30 03:29:23.040029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.040004    4278 scope.go:122] "RemoveContainer" containerID="45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55"
Oct 30 03:29:23.040374 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:29:23.040346    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55\": container with ID starting with 45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55 not found: ID does not exist" containerID="45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55"
Oct 30 03:29:23.040510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.040384    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55"} err="failed to get container status \"45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55\": rpc error: code = NotFound desc = could not find container \"45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55\": container with ID starting with 45666b86d99fd78575e7c5de303612395b39c0bca9457345ae021a9996c4df55 not found: ID does not exist"
Oct 30 03:29:23.040510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.040405    4278 scope.go:122] "RemoveContainer" containerID="e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2"
Oct 30 03:29:23.050506 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.050481    4278 scope.go:122] "RemoveContainer" containerID="e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2"
Oct 30 03:29:23.050824 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:29:23.050804    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2\": container with ID starting with e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2 not found: ID does not exist" containerID="e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2"
Oct 30 03:29:23.050896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.050833    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2"} err="failed to get container status \"e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2\": rpc error: code = NotFound desc = could not find container \"e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2\": container with ID starting with e077b1c3ffea500b6d1cdbb23796bd8f7bc206daf590bf42097f368e13d9d6f2 not found: ID does not exist"
Oct 30 03:29:23.088052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.088018    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"]
Oct 30 03:29:23.093642 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.093609    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/simulator-w6-32bad3-kserve-94fd4b9d-tbzfz"]
Oct 30 03:29:23.109216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.109188    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4584e9e-386b-47a3-93e3-abe7a1961f4c-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.109216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.109213    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.109216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.109221    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.109452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.109230    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jjp\" (UniqueName: \"kubernetes.io/projected/b4584e9e-386b-47a3-93e3-abe7a1961f4c-kube-api-access-q6jjp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.109452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.109240    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.109452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.109251    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4584e9e-386b-47a3-93e3-abe7a1961f4c-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.109452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.109259    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4bsv\" (UniqueName: \"kubernetes.io/projected/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-kube-api-access-k4bsv\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.109452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.109267    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b215a820-a1ac-49d0-9ad5-cd8a41b54d15-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.128199 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.128167    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"]
Oct 30 03:29:23.131199 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:23.131171    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-2-w6-32bad3-kserve-69856dd9-kmkfj"]
Oct 30 03:29:24.038659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:24.038559    4278 generic.go:356] "Generic (PLEG): container finished" podID="8bef6098-76d0-4f91-9ea8-8c80652f9ce6" containerID="145f57026ea3ed716c45ee4386ceb216c07fe44454cdf3bf89167ac028036d41" exitCode=0
Oct 30 03:29:24.038659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:24.038614    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh" event={"ID":"8bef6098-76d0-4f91-9ea8-8c80652f9ce6","Type":"ContainerDied","Data":"145f57026ea3ed716c45ee4386ceb216c07fe44454cdf3bf89167ac028036d41"}
Oct 30 03:29:24.190193 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:29:24.190148    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:29:24.199100 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:24.199062    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" path="/var/lib/kubelet/pods/b215a820-a1ac-49d0-9ad5-cd8a41b54d15/volumes"
Oct 30 03:29:24.199718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:24.199692    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" path="/var/lib/kubelet/pods/b4584e9e-386b-47a3-93e3-abe7a1961f4c/volumes"
Oct 30 03:29:24.947861 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:24.947815    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf" podUID="20c80749-612c-4782-82fe-4746cc8b920c" containerName="payload-processing" containerID="cri-o://2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa" gracePeriod=25
Oct 30 03:29:25.012011 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.011961    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt" podUID="386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6" containerName="payload-pre-processing" containerID="cri-o://c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9" gracePeriod=25
Oct 30 03:29:25.109860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.109812    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"]
Oct 30 03:29:25.110385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.110367    4278 cpu_manager.go:414] "containerMap: removing container" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main"
Oct 30 03:29:25.110385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.110385    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main"
Oct 30 03:29:25.110523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.110416    4278 cpu_manager.go:414] "containerMap: removing container" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main"
Oct 30 03:29:25.110523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.110435    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main"
Oct 30 03:29:25.110523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.110516    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4584e9e-386b-47a3-93e3-abe7a1961f4c" containerName="main"
Oct 30 03:29:25.110630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.110528    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="b215a820-a1ac-49d0-9ad5-cd8a41b54d15" containerName="main"
Oct 30 03:29:25.114377 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.113936    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.140717 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.134997    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"]
Oct 30 03:29:25.145213 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.145173    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"]
Oct 30 03:29:25.145736 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.145506    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.176957 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.176920    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"]
Oct 30 03:29:25.231021 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.230993    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:25.237302 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.237268    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw4pl\" (UniqueName: \"kubernetes.io/projected/ab139dfe-8de9-4fad-864d-3781187c1238-kube-api-access-xw4pl\") pod \"maas-api-revoke-sub-31bcf3c4-tpsdm\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.237673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.237646    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89jpl\" (UniqueName: \"kubernetes.io/projected/1e35e402-7c30-4cad-879e-817b569141ef-kube-api-access-89jpl\") pod \"maas-api-revoke-sub-2c640902-qjbb4\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.237928 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.237903    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c640902-qjbb4\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.238143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.238123    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c640902-qjbb4\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.238325 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.238306    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31bcf3c4-tpsdm\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.238495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.238478    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31bcf3c4-tpsdm\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.312001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.311974    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:29:25.339548 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.339514    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-xks-ca\") pod \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") "
Oct 30 03:29:25.339726 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.339645    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-openshift-ca\") pod \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") "
Oct 30 03:29:25.339726 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.339692    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-kube-api-access-rfs5d\" (UniqueName: \"kubernetes.io/projected/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-kube-api-access-rfs5d\") pod \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\" (UID: \"8bef6098-76d0-4f91-9ea8-8c80652f9ce6\") "
Oct 30 03:29:25.339896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.339874    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31bcf3c4-tpsdm\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.339973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.339948    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31bcf3c4-tpsdm\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.340024 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.339993    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-openshift-ca" pod "8bef6098-76d0-4f91-9ea8-8c80652f9ce6" (UID: "8bef6098-76d0-4f91-9ea8-8c80652f9ce6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:25.340138 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.340117    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw4pl\" (UniqueName: \"kubernetes.io/projected/ab139dfe-8de9-4fad-864d-3781187c1238-kube-api-access-xw4pl\") pod \"maas-api-revoke-sub-31bcf3c4-tpsdm\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.340223 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.340173    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89jpl\" (UniqueName: \"kubernetes.io/projected/1e35e402-7c30-4cad-879e-817b569141ef-kube-api-access-89jpl\") pod \"maas-api-revoke-sub-2c640902-qjbb4\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.340283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.340253    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c640902-qjbb4\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.340367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.340346    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c640902-qjbb4\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.340543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.340523    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:25.341059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.340874    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31bcf3c4-tpsdm\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.341059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.340894    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:29:25.341059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.341028    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c640902-qjbb4\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.342267 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.342234    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-kube-api-access-rfs5d" pod "8bef6098-76d0-4f91-9ea8-8c80652f9ce6" (UID: "8bef6098-76d0-4f91-9ea8-8c80652f9ce6"). InnerVolumeSpecName "kube-api-access-rfs5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:25.342757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.342736    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-xks-ca" pod "8bef6098-76d0-4f91-9ea8-8c80652f9ce6" (UID: "8bef6098-76d0-4f91-9ea8-8c80652f9ce6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:25.342942 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.342920    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31bcf3c4-tpsdm\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.343358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.343338    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c640902-qjbb4\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.359033 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.359005    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89jpl\" (UniqueName: \"kubernetes.io/projected/1e35e402-7c30-4cad-879e-817b569141ef-kube-api-access-89jpl\") pod \"maas-api-revoke-sub-2c640902-qjbb4\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.360297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.360271    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw4pl\" (UniqueName: \"kubernetes.io/projected/ab139dfe-8de9-4fad-864d-3781187c1238-kube-api-access-xw4pl\") pod \"maas-api-revoke-sub-31bcf3c4-tpsdm\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.441096 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.441055    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c80749-612c-4782-82fe-4746cc8b920c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c80749-612c-4782-82fe-4746cc8b920c-plugins-config-volume\") pod \"20c80749-612c-4782-82fe-4746cc8b920c\" (UID: \"20c80749-612c-4782-82fe-4746cc8b920c\") "
Oct 30 03:29:25.441278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.441167    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-plugins-config-volume\") pod \"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6\" (UID: \"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6\") "
Oct 30 03:29:25.441278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.441210    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-kube-api-access-cs8qv\" (UniqueName: \"kubernetes.io/projected/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-kube-api-access-cs8qv\") pod \"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6\" (UID: \"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6\") "
Oct 30 03:29:25.441406 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.441390    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c80749-612c-4782-82fe-4746cc8b920c-kube-api-access-zgpx7\" (UniqueName: \"kubernetes.io/projected/20c80749-612c-4782-82fe-4746cc8b920c-kube-api-access-zgpx7\") pod \"20c80749-612c-4782-82fe-4746cc8b920c\" (UID: \"20c80749-612c-4782-82fe-4746cc8b920c\") "
Oct 30 03:29:25.441542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.441522    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c80749-612c-4782-82fe-4746cc8b920c-plugins-config-volume" pod "20c80749-612c-4782-82fe-4746cc8b920c" (UID: "20c80749-612c-4782-82fe-4746cc8b920c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:25.441597 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.441555    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-plugins-config-volume" pod "386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6" (UID: "386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:25.441885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.441867    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c80749-612c-4782-82fe-4746cc8b920c-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:25.441929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.441889    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfs5d\" (UniqueName: \"kubernetes.io/projected/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-kube-api-access-rfs5d\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:25.441929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.441899    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:25.441929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.441909    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bef6098-76d0-4f91-9ea8-8c80652f9ce6-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:25.443497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.443477    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c80749-612c-4782-82fe-4746cc8b920c-kube-api-access-zgpx7" pod "20c80749-612c-4782-82fe-4746cc8b920c" (UID: "20c80749-612c-4782-82fe-4746cc8b920c"). InnerVolumeSpecName "kube-api-access-zgpx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:25.443566 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.443506    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-kube-api-access-cs8qv" pod "386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6" (UID: "386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6"). InnerVolumeSpecName "kube-api-access-cs8qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:25.452656 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.452627    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:25.470585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.470564    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:25.543330 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.543291    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs8qv\" (UniqueName: \"kubernetes.io/projected/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6-kube-api-access-cs8qv\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:25.543509 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.543331    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgpx7\" (UniqueName: \"kubernetes.io/projected/20c80749-612c-4782-82fe-4746cc8b920c-kube-api-access-zgpx7\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:25.758532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.758504    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"]
Oct 30 03:29:25.761193 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:29:25.761147    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab139dfe_8de9_4fad_864d_3781187c1238.slice/crio-e1bbff667a7f2b28a0b5dbf938eabeec5a3d4d28ff8cbc630ed4ffc665b62777 WatchSource:0}: Error finding container e1bbff667a7f2b28a0b5dbf938eabeec5a3d4d28ff8cbc630ed4ffc665b62777: Status 404 returned error can't find the container with id e1bbff667a7f2b28a0b5dbf938eabeec5a3d4d28ff8cbc630ed4ffc665b62777
Oct 30 03:29:25.784963 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:25.784933    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"]
Oct 30 03:29:25.789696 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:29:25.789672    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e35e402_7c30_4cad_879e_817b569141ef.slice/crio-4d9d21a318fd07c7ed80ae6655415ae889b8dedf9785c91e68ef60aa1fb20d2d WatchSource:0}: Error finding container 4d9d21a318fd07c7ed80ae6655415ae889b8dedf9785c91e68ef60aa1fb20d2d: Status 404 returned error can't find the container with id 4d9d21a318fd07c7ed80ae6655415ae889b8dedf9785c91e68ef60aa1fb20d2d
Oct 30 03:29:26.063392 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.063353    4278 generic.go:356] "Generic (PLEG): container finished" podID="386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6" containerID="c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9" exitCode=0
Oct 30 03:29:26.063673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.063399    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt" event={"ID":"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6","Type":"ContainerDied","Data":"c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9"}
Oct 30 03:29:26.063673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.063446    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"
Oct 30 03:29:26.063673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.063484    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt" event={"ID":"386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6","Type":"ContainerDied","Data":"44ec85a61585b524909c35764d07ad9fe2bdb020b26fd5f438286529f5e6c25d"}
Oct 30 03:29:26.063673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.063521    4278 scope.go:122] "RemoveContainer" containerID="c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9"
Oct 30 03:29:26.065476 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.065416    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4" event={"ID":"1e35e402-7c30-4cad-879e-817b569141ef","Type":"ContainerStarted","Data":"4d9d21a318fd07c7ed80ae6655415ae889b8dedf9785c91e68ef60aa1fb20d2d"}
Oct 30 03:29:26.067165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.067141    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"
Oct 30 03:29:26.067304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.067160    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh" event={"ID":"8bef6098-76d0-4f91-9ea8-8c80652f9ce6","Type":"ContainerDied","Data":"9bd1a35f1572ede887bb670e3bc2a60d7f2e51159ac2a3fe70c93605510b9abd"}
Oct 30 03:29:26.067304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.067201    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bd1a35f1572ede887bb670e3bc2a60d7f2e51159ac2a3fe70c93605510b9abd"
Oct 30 03:29:26.076802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.075047    4278 generic.go:356] "Generic (PLEG): container finished" podID="20c80749-612c-4782-82fe-4746cc8b920c" containerID="2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa" exitCode=0
Oct 30 03:29:26.076802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.075179    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf" event={"ID":"20c80749-612c-4782-82fe-4746cc8b920c","Type":"ContainerDied","Data":"2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa"}
Oct 30 03:29:26.076802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.075241    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf" event={"ID":"20c80749-612c-4782-82fe-4746cc8b920c","Type":"ContainerDied","Data":"fccb48373836b139e6c5620f95725e2094d4f0935ba764c542b8dad6c0d898e9"}
Oct 30 03:29:26.076802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.075341    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"
Oct 30 03:29:26.079167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.079137    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm" event={"ID":"ab139dfe-8de9-4fad-864d-3781187c1238","Type":"ContainerStarted","Data":"e1bbff667a7f2b28a0b5dbf938eabeec5a3d4d28ff8cbc630ed4ffc665b62777"}
Oct 30 03:29:26.082806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.082787    4278 scope.go:122] "RemoveContainer" containerID="c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9"
Oct 30 03:29:26.083143 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:29:26.083122    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9\": container with ID starting with c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9 not found: ID does not exist" containerID="c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9"
Oct 30 03:29:26.083244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.083149    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9"} err="failed to get container status \"c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9\": rpc error: code = NotFound desc = could not find container \"c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9\": container with ID starting with c7e9e94c83bf3a138dea562d39efbd28f25e9405274959c3ef652e71561c12c9 not found: ID does not exist"
Oct 30 03:29:26.083244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.083166    4278 scope.go:122] "RemoveContainer" containerID="2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa"
Oct 30 03:29:26.093620 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.093598    4278 scope.go:122] "RemoveContainer" containerID="2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa"
Oct 30 03:29:26.093959 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:29:26.093939    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa\": container with ID starting with 2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa not found: ID does not exist" containerID="2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa"
Oct 30 03:29:26.094015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.093968    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa"} err="failed to get container status \"2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa\": rpc error: code = NotFound desc = could not find container \"2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa\": container with ID starting with 2d9769f9aea68a0c0793217744670e59227daf360d74164400917952ffc1e3aa not found: ID does not exist"
Oct 30 03:29:26.127994 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.127961    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"]
Oct 30 03:29:26.131078 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.131048    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-6bf8d76564-mm2wf"]
Oct 30 03:29:26.162843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.162810    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"]
Oct 30 03:29:26.165972 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.165945    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-669b745cf9-v94mt"]
Oct 30 03:29:26.199419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.199381    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c80749-612c-4782-82fe-4746cc8b920c" path="/var/lib/kubelet/pods/20c80749-612c-4782-82fe-4746cc8b920c/volumes"
Oct 30 03:29:26.199932 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:26.199904    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6" path="/var/lib/kubelet/pods/386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6/volumes"
Oct 30 03:29:27.089855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:27.089817    4278 generic.go:356] "Generic (PLEG): container finished" podID="1e35e402-7c30-4cad-879e-817b569141ef" containerID="33ce457a6b11c252feab7a56face5414bb6bcfe2ff2b4c8c8e95551ddd3c1cfc" exitCode=0
Oct 30 03:29:27.090080 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:27.089900    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4" event={"ID":"1e35e402-7c30-4cad-879e-817b569141ef","Type":"ContainerDied","Data":"33ce457a6b11c252feab7a56face5414bb6bcfe2ff2b4c8c8e95551ddd3c1cfc"}
Oct 30 03:29:27.092681 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:27.092651    4278 generic.go:356] "Generic (PLEG): container finished" podID="ab139dfe-8de9-4fad-864d-3781187c1238" containerID="7133d718c55166c16e8306ff38f550576c9aff7c2c7e74106e3bfc87845becb0" exitCode=0
Oct 30 03:29:27.092817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:27.092721    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm" event={"ID":"ab139dfe-8de9-4fad-864d-3781187c1238","Type":"ContainerDied","Data":"7133d718c55166c16e8306ff38f550576c9aff7c2c7e74106e3bfc87845becb0"}
Oct 30 03:29:28.288373 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.288343    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:28.291523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.291497    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:28.375945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.375857    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e35e402-7c30-4cad-879e-817b569141ef-kube-api-access-89jpl\" (UniqueName: \"kubernetes.io/projected/1e35e402-7c30-4cad-879e-817b569141ef-kube-api-access-89jpl\") pod \"1e35e402-7c30-4cad-879e-817b569141ef\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") "
Oct 30 03:29:28.375945 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.375913    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-xks-ca\") pod \"ab139dfe-8de9-4fad-864d-3781187c1238\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") "
Oct 30 03:29:28.376152 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.376039    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab139dfe-8de9-4fad-864d-3781187c1238-kube-api-access-xw4pl\" (UniqueName: \"kubernetes.io/projected/ab139dfe-8de9-4fad-864d-3781187c1238-kube-api-access-xw4pl\") pod \"ab139dfe-8de9-4fad-864d-3781187c1238\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") "
Oct 30 03:29:28.376152 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.376089    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-openshift-ca\") pod \"1e35e402-7c30-4cad-879e-817b569141ef\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") "
Oct 30 03:29:28.376152 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.376144    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-xks-ca\") pod \"1e35e402-7c30-4cad-879e-817b569141ef\" (UID: \"1e35e402-7c30-4cad-879e-817b569141ef\") "
Oct 30 03:29:28.376318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.376203    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-openshift-ca\") pod \"ab139dfe-8de9-4fad-864d-3781187c1238\" (UID: \"ab139dfe-8de9-4fad-864d-3781187c1238\") "
Oct 30 03:29:28.376576 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.376550    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-openshift-ca" pod "1e35e402-7c30-4cad-879e-817b569141ef" (UID: "1e35e402-7c30-4cad-879e-817b569141ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:28.376839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.376781    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-openshift-ca" pod "ab139dfe-8de9-4fad-864d-3781187c1238" (UID: "ab139dfe-8de9-4fad-864d-3781187c1238"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:28.376839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.376787    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:28.378927 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.378903    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-xks-ca" pod "ab139dfe-8de9-4fad-864d-3781187c1238" (UID: "ab139dfe-8de9-4fad-864d-3781187c1238"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:28.378927 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.378920    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e35e402-7c30-4cad-879e-817b569141ef-kube-api-access-89jpl" pod "1e35e402-7c30-4cad-879e-817b569141ef" (UID: "1e35e402-7c30-4cad-879e-817b569141ef"). InnerVolumeSpecName "kube-api-access-89jpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:28.379149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.378898    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab139dfe-8de9-4fad-864d-3781187c1238-kube-api-access-xw4pl" pod "ab139dfe-8de9-4fad-864d-3781187c1238" (UID: "ab139dfe-8de9-4fad-864d-3781187c1238"). InnerVolumeSpecName "kube-api-access-xw4pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:28.379149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.378944    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-xks-ca" pod "1e35e402-7c30-4cad-879e-817b569141ef" (UID: "1e35e402-7c30-4cad-879e-817b569141ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:28.478088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.478042    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw4pl\" (UniqueName: \"kubernetes.io/projected/ab139dfe-8de9-4fad-864d-3781187c1238-kube-api-access-xw4pl\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:28.478088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.478077    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e35e402-7c30-4cad-879e-817b569141ef-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:28.478088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.478088    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:28.478088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.478097    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89jpl\" (UniqueName: \"kubernetes.io/projected/1e35e402-7c30-4cad-879e-817b569141ef-kube-api-access-89jpl\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:28.478383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:28.478106    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab139dfe-8de9-4fad-864d-3781187c1238-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:29.107706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:29.107668    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"
Oct 30 03:29:29.107706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:29.107682    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4" event={"ID":"1e35e402-7c30-4cad-879e-817b569141ef","Type":"ContainerDied","Data":"4d9d21a318fd07c7ed80ae6655415ae889b8dedf9785c91e68ef60aa1fb20d2d"}
Oct 30 03:29:29.107948 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:29.107765    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d9d21a318fd07c7ed80ae6655415ae889b8dedf9785c91e68ef60aa1fb20d2d"
Oct 30 03:29:29.109240 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:29.109220    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"
Oct 30 03:29:29.109349 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:29.109250    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm" event={"ID":"ab139dfe-8de9-4fad-864d-3781187c1238","Type":"ContainerDied","Data":"e1bbff667a7f2b28a0b5dbf938eabeec5a3d4d28ff8cbc630ed4ffc665b62777"}
Oct 30 03:29:29.109349 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:29.109298    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1bbff667a7f2b28a0b5dbf938eabeec5a3d4d28ff8cbc630ed4ffc665b62777"
Oct 30 03:29:29.324237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:29.324201    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:29:38.509299 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:38.509257    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"]
Oct 30 03:29:38.520294 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:38.520249    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b5f19-lqdrh"]
Oct 30 03:29:40.026021 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:40.025980    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"]
Oct 30 03:29:40.037344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:40.037305    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02df6932-hbdh7"]
Oct 30 03:29:40.199079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:40.199045    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bef6098-76d0-4f91-9ea8-8c80652f9ce6" path="/var/lib/kubelet/pods/8bef6098-76d0-4f91-9ea8-8c80652f9ce6/volumes"
Oct 30 03:29:40.199557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:40.199541    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a62072-3a62-4e1a-b33b-7a198b372c04" path="/var/lib/kubelet/pods/c4a62072-3a62-4e1a-b33b-7a198b372c04/volumes"
Oct 30 03:29:41.025879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:41.025839    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"]
Oct 30 03:29:41.036726 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:41.036695    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ee4798-zr2xl"]
Oct 30 03:29:42.198502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:42.198463    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="838a3dda-fd7f-404b-af80-0cf69627af89" path="/var/lib/kubelet/pods/838a3dda-fd7f-404b-af80-0cf69627af89/volumes"
Oct 30 03:29:45.090603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:45.090506    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"]
Oct 30 03:29:45.101759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:45.101724    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-32bad3-6f5xj"]
Oct 30 03:29:45.186695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:45.186650    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:29:46.198288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:46.198243    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc" path="/var/lib/kubelet/pods/a9d0ddf1-eafd-4cab-a877-d4d6848ca7bc/volumes"
Oct 30 03:29:47.063896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.063853    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"]
Oct 30 03:29:47.064581 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064557    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8bef6098-76d0-4f91-9ea8-8c80652f9ce6" containerName="revoke-keys"
Oct 30 03:29:47.064704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064583    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bef6098-76d0-4f91-9ea8-8c80652f9ce6" containerName="revoke-keys"
Oct 30 03:29:47.064704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064623    4278 cpu_manager.go:414] "containerMap: removing container" podUID="1e35e402-7c30-4cad-879e-817b569141ef" containerName="revoke-keys"
Oct 30 03:29:47.064704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064631    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e35e402-7c30-4cad-879e-817b569141ef" containerName="revoke-keys"
Oct 30 03:29:47.064704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064652    4278 cpu_manager.go:414] "containerMap: removing container" podUID="20c80749-612c-4782-82fe-4746cc8b920c" containerName="payload-processing"
Oct 30 03:29:47.064704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064659    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c80749-612c-4782-82fe-4746cc8b920c" containerName="payload-processing"
Oct 30 03:29:47.064704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064670    4278 cpu_manager.go:414] "containerMap: removing container" podUID="386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6" containerName="payload-pre-processing"
Oct 30 03:29:47.064704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064679    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6" containerName="payload-pre-processing"
Oct 30 03:29:47.064704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064705    4278 cpu_manager.go:414] "containerMap: removing container" podUID="ab139dfe-8de9-4fad-864d-3781187c1238" containerName="revoke-keys"
Oct 30 03:29:47.065003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064713    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab139dfe-8de9-4fad-864d-3781187c1238" containerName="revoke-keys"
Oct 30 03:29:47.065003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064817    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab139dfe-8de9-4fad-864d-3781187c1238" containerName="revoke-keys"
Oct 30 03:29:47.065003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064833    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bef6098-76d0-4f91-9ea8-8c80652f9ce6" containerName="revoke-keys"
Oct 30 03:29:47.065003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064851    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e35e402-7c30-4cad-879e-817b569141ef" containerName="revoke-keys"
Oct 30 03:29:47.065003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064861    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c80749-612c-4782-82fe-4746cc8b920c" containerName="payload-processing"
Oct 30 03:29:47.065003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.064870    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="386cc0d6-6b0d-4a45-972e-d2a8ba88e9a6" containerName="payload-pre-processing"
Oct 30 03:29:47.069050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.068991    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.087700 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.087663    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"]
Oct 30 03:29:47.096048 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.096004    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:29:47.096235 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.096058    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:29:47.163329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.163292    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:29:47.163560 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.163406    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:29:47.211949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.211911    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.212398 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.211959    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gkcq\" (UniqueName: \"kubernetes.io/projected/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-kube-api-access-9gkcq\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.212398 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.212046    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.313822 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.313781    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.313822 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.313828    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gkcq\" (UniqueName: \"kubernetes.io/projected/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-kube-api-access-9gkcq\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.314167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.313922    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.314740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.314715    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.317039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.317014    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.332285 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.332250    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gkcq\" (UniqueName: \"kubernetes.io/projected/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-kube-api-access-9gkcq\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.385324 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.385280    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:47.694550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:47.694274    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"]
Oct 30 03:29:47.697199 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:29:47.697171    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ee9891_edd4_4c0e_b066_b52c9ef78e2a.slice/crio-5da78045d94ba9e1f3afdda78093c5f1f0306016e08de667035d135e30b9c387 WatchSource:0}: Error finding container 5da78045d94ba9e1f3afdda78093c5f1f0306016e08de667035d135e30b9c387: Status 404 returned error can't find the container with id 5da78045d94ba9e1f3afdda78093c5f1f0306016e08de667035d135e30b9c387
Oct 30 03:29:48.256661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:48.256620    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp" event={"ID":"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a","Type":"ContainerStarted","Data":"a7a1e54e84375ce54630bb9ea7275fd821412719d89dd0d5b62bd4572d37cdf4"}
Oct 30 03:29:48.257141 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:48.256682    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp" event={"ID":"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a","Type":"ContainerStarted","Data":"5da78045d94ba9e1f3afdda78093c5f1f0306016e08de667035d135e30b9c387"}
Oct 30 03:29:48.305689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:48.305632    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp" podStartSLOduration=1.3056166280000001 podStartE2EDuration="1.305616628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:48.293411378 +0000 UTC m=+2926.217619568" watchObservedRunningTime="2026-10-30 03:29:48.305616628 +0000 UTC m=+2926.229824816"
Oct 30 03:29:49.265454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:49.265398    4278 generic.go:356] "Generic (PLEG): container finished" podID="c0ee9891-edd4-4c0e-b066-b52c9ef78e2a" containerID="a7a1e54e84375ce54630bb9ea7275fd821412719d89dd0d5b62bd4572d37cdf4" exitCode=0
Oct 30 03:29:49.265874 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:49.265457    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp" event={"ID":"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a","Type":"ContainerDied","Data":"a7a1e54e84375ce54630bb9ea7275fd821412719d89dd0d5b62bd4572d37cdf4"}
Oct 30 03:29:50.416713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:50.416685    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:50.449760 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:50.449725    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-xks-ca\") pod \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") "
Oct 30 03:29:50.449923 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:50.449776    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-kube-api-access-9gkcq\" (UniqueName: \"kubernetes.io/projected/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-kube-api-access-9gkcq\") pod \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") "
Oct 30 03:29:50.449923 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:50.449837    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-openshift-ca\") pod \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\" (UID: \"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a\") "
Oct 30 03:29:50.450284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:50.450255    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-openshift-ca" pod "c0ee9891-edd4-4c0e-b066-b52c9ef78e2a" (UID: "c0ee9891-edd4-4c0e-b066-b52c9ef78e2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:50.452034 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:50.452005    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-kube-api-access-9gkcq" pod "c0ee9891-edd4-4c0e-b066-b52c9ef78e2a" (UID: "c0ee9891-edd4-4c0e-b066-b52c9ef78e2a"). InnerVolumeSpecName "kube-api-access-9gkcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:50.452149 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:50.452048    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-xks-ca" pod "c0ee9891-edd4-4c0e-b066-b52c9ef78e2a" (UID: "c0ee9891-edd4-4c0e-b066-b52c9ef78e2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:50.551206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:50.551118    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gkcq\" (UniqueName: \"kubernetes.io/projected/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-kube-api-access-9gkcq\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:50.551206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:50.551150    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:50.551206 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:50.551161    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:51.280839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:51.280801    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"
Oct 30 03:29:51.280839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:51.280826    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp" event={"ID":"c0ee9891-edd4-4c0e-b066-b52c9ef78e2a","Type":"ContainerDied","Data":"5da78045d94ba9e1f3afdda78093c5f1f0306016e08de667035d135e30b9c387"}
Oct 30 03:29:51.281068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:51.280880    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5da78045d94ba9e1f3afdda78093c5f1f0306016e08de667035d135e30b9c387"
Oct 30 03:29:51.769028 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:51.768986    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"]
Oct 30 03:29:51.769754 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:51.769266    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" podUID="54319ced-9220-46fb-a97e-6c0b1e702d50" containerName="istio-proxy" containerID="cri-o://910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199" gracePeriod=30
Oct 30 03:29:54.189692 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:29:54.189639    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:29:57.038374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.038339    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 03:29:57.131338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131245    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-envoy\") pod \"54319ced-9220-46fb-a97e-6c0b1e702d50\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") "
Oct 30 03:29:57.131564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131338    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-credential-socket\") pod \"54319ced-9220-46fb-a97e-6c0b1e702d50\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") "
Oct 30 03:29:57.131564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131372    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-certs\") pod \"54319ced-9220-46fb-a97e-6c0b1e702d50\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") "
Oct 30 03:29:57.131564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131409    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-podinfo\") pod \"54319ced-9220-46fb-a97e-6c0b1e702d50\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") "
Oct 30 03:29:57.131564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131506    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-kube-api-access-tznc2\" (UniqueName: \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-kube-api-access-tznc2\") pod \"54319ced-9220-46fb-a97e-6c0b1e702d50\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") "
Oct 30 03:29:57.131564 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131551    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54319ced-9220-46fb-a97e-6c0b1e702d50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54319ced-9220-46fb-a97e-6c0b1e702d50-istiod-ca-cert\") pod \"54319ced-9220-46fb-a97e-6c0b1e702d50\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") "
Oct 30 03:29:57.131870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131633    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-data\") pod \"54319ced-9220-46fb-a97e-6c0b1e702d50\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") "
Oct 30 03:29:57.131870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131793    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-certs" pod "54319ced-9220-46fb-a97e-6c0b1e702d50" (UID: "54319ced-9220-46fb-a97e-6c0b1e702d50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:57.131870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131844    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-credential-socket" pod "54319ced-9220-46fb-a97e-6c0b1e702d50" (UID: "54319ced-9220-46fb-a97e-6c0b1e702d50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:57.131870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131865    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-socket\") pod \"54319ced-9220-46fb-a97e-6c0b1e702d50\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") "
Oct 30 03:29:57.132094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.131959    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-token\" (UniqueName: \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-token\") pod \"54319ced-9220-46fb-a97e-6c0b1e702d50\" (UID: \"54319ced-9220-46fb-a97e-6c0b1e702d50\") "
Oct 30 03:29:57.132144 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.132110    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-socket" pod "54319ced-9220-46fb-a97e-6c0b1e702d50" (UID: "54319ced-9220-46fb-a97e-6c0b1e702d50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:57.132199 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.132141    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54319ced-9220-46fb-a97e-6c0b1e702d50-istiod-ca-cert" pod "54319ced-9220-46fb-a97e-6c0b1e702d50" (UID: "54319ced-9220-46fb-a97e-6c0b1e702d50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:57.132407 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.132354    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-data" pod "54319ced-9220-46fb-a97e-6c0b1e702d50" (UID: "54319ced-9220-46fb-a97e-6c0b1e702d50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:57.132534 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.132512    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.132607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.132541    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54319ced-9220-46fb-a97e-6c0b1e702d50-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.132607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.132556    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.132607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.132570    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.132607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.132583    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.133773 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.133729    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-envoy" pod "54319ced-9220-46fb-a97e-6c0b1e702d50" (UID: "54319ced-9220-46fb-a97e-6c0b1e702d50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:57.133899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.133827    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-podinfo" pod "54319ced-9220-46fb-a97e-6c0b1e702d50" (UID: "54319ced-9220-46fb-a97e-6c0b1e702d50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:29:57.134291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.134263    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-kube-api-access-tznc2" pod "54319ced-9220-46fb-a97e-6c0b1e702d50" (UID: "54319ced-9220-46fb-a97e-6c0b1e702d50"). InnerVolumeSpecName "kube-api-access-tznc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:57.134390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.134372    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-token" pod "54319ced-9220-46fb-a97e-6c0b1e702d50" (UID: "54319ced-9220-46fb-a97e-6c0b1e702d50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:57.233618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.233574    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.233618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.233606    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.233618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.233618    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tznc2\" (UniqueName: \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-kube-api-access-tznc2\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.233618 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.233629    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54319ced-9220-46fb-a97e-6c0b1e702d50-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.321788 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.321745    4278 generic.go:356] "Generic (PLEG): container finished" podID="54319ced-9220-46fb-a97e-6c0b1e702d50" containerID="910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199" exitCode=0
Oct 30 03:29:57.322004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.321804    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"
Oct 30 03:29:57.322004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.321842    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" event={"ID":"54319ced-9220-46fb-a97e-6c0b1e702d50","Type":"ContainerDied","Data":"910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199"}
Oct 30 03:29:57.322004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.321914    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f" event={"ID":"54319ced-9220-46fb-a97e-6c0b1e702d50","Type":"ContainerDied","Data":"4186d5ba7ab30ad54430f0964b427527107859911a4dbb858e808b4637641780"}
Oct 30 03:29:57.322004 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.321947    4278 scope.go:122] "RemoveContainer" containerID="910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199"
Oct 30 03:29:57.333420 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.333397    4278 scope.go:122] "RemoveContainer" containerID="910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199"
Oct 30 03:29:57.333757 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:29:57.333724    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199\": container with ID starting with 910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199 not found: ID does not exist" containerID="910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199"
Oct 30 03:29:57.333843 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.333760    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199"} err="failed to get container status \"910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199\": rpc error: code = NotFound desc = could not find container \"910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199\": container with ID starting with 910b0bf3b80c7060260a9044a06d991f3d2f3854619cc31151b866d84d84b199 not found: ID does not exist"
Oct 30 03:29:57.372177 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.372142    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"]
Oct 30 03:29:57.383145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:57.383055    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-b4c9f"]
Oct 30 03:29:58.199189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:58.199151    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54319ced-9220-46fb-a97e-6c0b1e702d50" path="/var/lib/kubelet/pods/54319ced-9220-46fb-a97e-6c0b1e702d50/volumes"
Oct 30 03:29:59.282327 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:29:59.282289    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:30:00.152456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.152379    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"]
Oct 30 03:30:00.153367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.153335    4278 cpu_manager.go:414] "containerMap: removing container" podUID="54319ced-9220-46fb-a97e-6c0b1e702d50" containerName="istio-proxy"
Oct 30 03:30:00.153367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.153363    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54319ced-9220-46fb-a97e-6c0b1e702d50" containerName="istio-proxy"
Oct 30 03:30:00.153663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.153387    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c0ee9891-edd4-4c0e-b066-b52c9ef78e2a" containerName="revoke-keys"
Oct 30 03:30:00.153663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.153396    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ee9891-edd4-4c0e-b066-b52c9ef78e2a" containerName="revoke-keys"
Oct 30 03:30:00.153663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.153521    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="54319ced-9220-46fb-a97e-6c0b1e702d50" containerName="istio-proxy"
Oct 30 03:30:00.153663 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.153539    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ee9891-edd4-4c0e-b066-b52c9ef78e2a" containerName="revoke-keys"
Oct 30 03:30:00.158962 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.158923    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"]
Oct 30 03:30:00.159131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.159067    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"
Oct 30 03:30:00.162905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.162875    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"]
Oct 30 03:30:00.163073 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.162985    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"
Oct 30 03:30:00.166914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.166890    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"]
Oct 30 03:30:00.167070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.167006    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"
Oct 30 03:30:00.171291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.171259    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"
Oct 30 03:30:00.177714 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.177671    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:30:00.178290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.178257    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"]
Oct 30 03:30:00.179874 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.179851    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:30:00.180132 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.180092    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:30:00.182773 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.182752    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"
Oct 30 03:30:00.191167 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.191138    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"]
Oct 30 03:30:00.206776 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.206739    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"]
Oct 30 03:30:00.206982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.206812    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"]
Oct 30 03:30:00.206982 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.206843    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"]
Oct 30 03:30:00.231272 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.231226    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"]
Oct 30 03:30:00.267848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.267815    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8v8x\" (UniqueName: \"kubernetes.io/projected/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-kube-api-access-c8v8x\") pod \"maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq\" (UID: \"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"
Oct 30 03:30:00.268046 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.267872    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqw9c\" (UniqueName: \"kubernetes.io/projected/fe31905f-5f69-451b-95c6-dd619b33cb29-kube-api-access-wqw9c\") pod \"maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf\" (UID: \"fe31905f-5f69-451b-95c6-dd619b33cb29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"
Oct 30 03:30:00.268046 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.267955    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba040b1-30f4-49c5-9168-a41ddf161956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm\" (UID: \"5ba040b1-30f4-49c5-9168-a41ddf161956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"
Oct 30 03:30:00.268135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.268057    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88qz\" (UniqueName: \"kubernetes.io/projected/5ba040b1-30f4-49c5-9168-a41ddf161956-kube-api-access-b88qz\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm\" (UID: \"5ba040b1-30f4-49c5-9168-a41ddf161956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"
Oct 30 03:30:00.268135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.268110    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvklh\" (UniqueName: \"kubernetes.io/projected/212e361c-7a49-4276-be50-34407c4bd146-kube-api-access-xvklh\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl\" (UID: \"212e361c-7a49-4276-be50-34407c4bd146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"
Oct 30 03:30:00.268218 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.268147    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe31905f-5f69-451b-95c6-dd619b33cb29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf\" (UID: \"fe31905f-5f69-451b-95c6-dd619b33cb29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"
Oct 30 03:30:00.268268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.268255    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212e361c-7a49-4276-be50-34407c4bd146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl\" (UID: \"212e361c-7a49-4276-be50-34407c4bd146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"
Oct 30 03:30:00.268314 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.268285    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq\" (UID: \"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"
Oct 30 03:30:00.268350 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.268326    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037768be-210c-4930-a922-a1cfd9e8137e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs\" (UID: \"037768be-210c-4930-a922-a1cfd9e8137e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"
Oct 30 03:30:00.268394 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.268346    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5k67\" (UniqueName: \"kubernetes.io/projected/037768be-210c-4930-a922-a1cfd9e8137e-kube-api-access-c5k67\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs\" (UID: \"037768be-210c-4930-a922-a1cfd9e8137e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"
Oct 30 03:30:00.369654 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.369604    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvklh\" (UniqueName: \"kubernetes.io/projected/212e361c-7a49-4276-be50-34407c4bd146-kube-api-access-xvklh\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl\" (UID: \"212e361c-7a49-4276-be50-34407c4bd146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"
Oct 30 03:30:00.370083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.369698    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe31905f-5f69-451b-95c6-dd619b33cb29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf\" (UID: \"fe31905f-5f69-451b-95c6-dd619b33cb29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"
Oct 30 03:30:00.370083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.369740    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212e361c-7a49-4276-be50-34407c4bd146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl\" (UID: \"212e361c-7a49-4276-be50-34407c4bd146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"
Oct 30 03:30:00.370083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.369778    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq\" (UID: \"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"
Oct 30 03:30:00.370083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.369811    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037768be-210c-4930-a922-a1cfd9e8137e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs\" (UID: \"037768be-210c-4930-a922-a1cfd9e8137e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"
Oct 30 03:30:00.370083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.369836    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5k67\" (UniqueName: \"kubernetes.io/projected/037768be-210c-4930-a922-a1cfd9e8137e-kube-api-access-c5k67\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs\" (UID: \"037768be-210c-4930-a922-a1cfd9e8137e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"
Oct 30 03:30:00.370083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.369910    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8v8x\" (UniqueName: \"kubernetes.io/projected/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-kube-api-access-c8v8x\") pod \"maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq\" (UID: \"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"
Oct 30 03:30:00.370083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.369973    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqw9c\" (UniqueName: \"kubernetes.io/projected/fe31905f-5f69-451b-95c6-dd619b33cb29-kube-api-access-wqw9c\") pod \"maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf\" (UID: \"fe31905f-5f69-451b-95c6-dd619b33cb29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"
Oct 30 03:30:00.370083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.369998    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba040b1-30f4-49c5-9168-a41ddf161956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm\" (UID: \"5ba040b1-30f4-49c5-9168-a41ddf161956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"
Oct 30 03:30:00.370083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.370070    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88qz\" (UniqueName: \"kubernetes.io/projected/5ba040b1-30f4-49c5-9168-a41ddf161956-kube-api-access-b88qz\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm\" (UID: \"5ba040b1-30f4-49c5-9168-a41ddf161956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"
Oct 30 03:30:00.370640 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.370620    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212e361c-7a49-4276-be50-34407c4bd146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl\" (UID: \"212e361c-7a49-4276-be50-34407c4bd146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"
Oct 30 03:30:00.370686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.370648    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe31905f-5f69-451b-95c6-dd619b33cb29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf\" (UID: \"fe31905f-5f69-451b-95c6-dd619b33cb29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"
Oct 30 03:30:00.370686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.370665    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq\" (UID: \"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"
Oct 30 03:30:00.370814 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.370799    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba040b1-30f4-49c5-9168-a41ddf161956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm\" (UID: \"5ba040b1-30f4-49c5-9168-a41ddf161956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"
Oct 30 03:30:00.371083 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.371065    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037768be-210c-4930-a922-a1cfd9e8137e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs\" (UID: \"037768be-210c-4930-a922-a1cfd9e8137e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"
Oct 30 03:30:00.389540 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.389506    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvklh\" (UniqueName: \"kubernetes.io/projected/212e361c-7a49-4276-be50-34407c4bd146-kube-api-access-xvklh\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl\" (UID: \"212e361c-7a49-4276-be50-34407c4bd146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"
Oct 30 03:30:00.389802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.389779    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqw9c\" (UniqueName: \"kubernetes.io/projected/fe31905f-5f69-451b-95c6-dd619b33cb29-kube-api-access-wqw9c\") pod \"maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf\" (UID: \"fe31905f-5f69-451b-95c6-dd619b33cb29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"
Oct 30 03:30:00.389896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.389865    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5k67\" (UniqueName: \"kubernetes.io/projected/037768be-210c-4930-a922-a1cfd9e8137e-kube-api-access-c5k67\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs\" (UID: \"037768be-210c-4930-a922-a1cfd9e8137e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"
Oct 30 03:30:00.389896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.389884    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8v8x\" (UniqueName: \"kubernetes.io/projected/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-kube-api-access-c8v8x\") pod \"maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq\" (UID: \"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"
Oct 30 03:30:00.390502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.390486    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88qz\" (UniqueName: \"kubernetes.io/projected/5ba040b1-30f4-49c5-9168-a41ddf161956-kube-api-access-b88qz\") pod \"maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm\" (UID: \"5ba040b1-30f4-49c5-9168-a41ddf161956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"
Oct 30 03:30:00.476561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.476465    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"
Oct 30 03:30:00.489997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.489972    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"
Oct 30 03:30:00.512300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.512258    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"
Oct 30 03:30:00.525877 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.525844    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"
Oct 30 03:30:00.536391 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.536358    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"
Oct 30 03:30:00.789412 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.789378    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"]
Oct 30 03:30:00.792263 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:30:00.792234    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba040b1_30f4_49c5_9168_a41ddf161956.slice/crio-ae25b8f5613af05f063ab2f58a556d2bf13e058b460a382d3816a1d0534c23a6 WatchSource:0}: Error finding container ae25b8f5613af05f063ab2f58a556d2bf13e058b460a382d3816a1d0534c23a6: Status 404 returned error can't find the container with id ae25b8f5613af05f063ab2f58a556d2bf13e058b460a382d3816a1d0534c23a6
Oct 30 03:30:00.801980 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.801954    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"]
Oct 30 03:30:00.806052 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:30:00.806025    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212e361c_7a49_4276_be50_34407c4bd146.slice/crio-1b3c61abe22dc1f7f594f336acf3365a8c714d9fead67e02bce1a9df32d64d0a WatchSource:0}: Error finding container 1b3c61abe22dc1f7f594f336acf3365a8c714d9fead67e02bce1a9df32d64d0a: Status 404 returned error can't find the container with id 1b3c61abe22dc1f7f594f336acf3365a8c714d9fead67e02bce1a9df32d64d0a
Oct 30 03:30:00.847142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.847065    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"]
Oct 30 03:30:00.851558 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:30:00.851528    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3baf589a_f6ff_4f1b_8402_e69d9bb0f98b.slice/crio-93296da079193ff158e16ad2c493f5784c890a49705d21873e15eeb5a037d81c WatchSource:0}: Error finding container 93296da079193ff158e16ad2c493f5784c890a49705d21873e15eeb5a037d81c: Status 404 returned error can't find the container with id 93296da079193ff158e16ad2c493f5784c890a49705d21873e15eeb5a037d81c
Oct 30 03:30:00.877032 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.876914    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"]
Oct 30 03:30:00.880213 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:30:00.880186    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037768be_210c_4930_a922_a1cfd9e8137e.slice/crio-bee3f04cc44b4ee0f2f4540bcb8d91ceb74a430a2335c20e197998ab126c1967 WatchSource:0}: Error finding container bee3f04cc44b4ee0f2f4540bcb8d91ceb74a430a2335c20e197998ab126c1967: Status 404 returned error can't find the container with id bee3f04cc44b4ee0f2f4540bcb8d91ceb74a430a2335c20e197998ab126c1967
Oct 30 03:30:00.908636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:00.908612    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"]
Oct 30 03:30:00.911025 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:30:00.910999    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe31905f_5f69_451b_95c6_dd619b33cb29.slice/crio-d8a7a59dda182692323804b602a17b845ac7e21f4addf88a91436326ece3a72e WatchSource:0}: Error finding container d8a7a59dda182692323804b602a17b845ac7e21f4addf88a91436326ece3a72e: Status 404 returned error can't find the container with id d8a7a59dda182692323804b602a17b845ac7e21f4addf88a91436326ece3a72e
Oct 30 03:30:01.351857 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.351753    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq" event={"ID":"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b","Type":"ContainerStarted","Data":"93296da079193ff158e16ad2c493f5784c890a49705d21873e15eeb5a037d81c"}
Oct 30 03:30:01.353127 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.353105    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" event={"ID":"037768be-210c-4930-a922-a1cfd9e8137e","Type":"ContainerStarted","Data":"bee3f04cc44b4ee0f2f4540bcb8d91ceb74a430a2335c20e197998ab126c1967"}
Oct 30 03:30:01.354837 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.354801    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" event={"ID":"fe31905f-5f69-451b-95c6-dd619b33cb29","Type":"ContainerStarted","Data":"89f956f60276f9ab46673dca2839edc4dc33fdf4a6d4dad99b7aad6f71e61ca6"}
Oct 30 03:30:01.354974 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.354868    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" event={"ID":"fe31905f-5f69-451b-95c6-dd619b33cb29","Type":"ContainerStarted","Data":"d8a7a59dda182692323804b602a17b845ac7e21f4addf88a91436326ece3a72e"}
Oct 30 03:30:01.356622 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.356592    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl" event={"ID":"212e361c-7a49-4276-be50-34407c4bd146","Type":"ContainerStarted","Data":"c2d5f8163701f59104dcdef5e0c9820efe6856f3e23824030b72f7dbe2d28b82"}
Oct 30 03:30:01.356738 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.356640    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl" event={"ID":"212e361c-7a49-4276-be50-34407c4bd146","Type":"ContainerStarted","Data":"1b3c61abe22dc1f7f594f336acf3365a8c714d9fead67e02bce1a9df32d64d0a"}
Oct 30 03:30:01.358328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.358301    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" event={"ID":"5ba040b1-30f4-49c5-9168-a41ddf161956","Type":"ContainerStarted","Data":"bbe36767b4c2a878787e30d893bc21b2f7aaa6884457500f38d567003eb366cf"}
Oct 30 03:30:01.358693 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.358353    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" event={"ID":"5ba040b1-30f4-49c5-9168-a41ddf161956","Type":"ContainerStarted","Data":"ae25b8f5613af05f063ab2f58a556d2bf13e058b460a382d3816a1d0534c23a6"}
Oct 30 03:30:01.392608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.392549    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" podStartSLOduration=1.392533941 podStartE2EDuration="1.392533941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.391370956 +0000 UTC m=+2939.315579145" watchObservedRunningTime="2026-10-30 03:30:01.392533941 +0000 UTC m=+2939.316742160"
Oct 30 03:30:01.429966 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.429902    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" podStartSLOduration=1.429887043 podStartE2EDuration="1.429887043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.426099859 +0000 UTC m=+2939.350308046" watchObservedRunningTime="2026-10-30 03:30:01.429887043 +0000 UTC m=+2939.354095230"
Oct 30 03:30:01.469239 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:01.469177    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl" podStartSLOduration=1.469161616 podStartE2EDuration="1.469161616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.466474346 +0000 UTC m=+2939.390682531" watchObservedRunningTime="2026-10-30 03:30:01.469161616 +0000 UTC m=+2939.393369803"
Oct 30 03:30:02.367234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.367191    4278 generic.go:356] "Generic (PLEG): container finished" podID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerID="71c8d1d862808bd2d3c4269bf38cdfe29ce72941f3b3996e242d27a70442f272" exitCode=60
Oct 30 03:30:02.367493 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.367272    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq" event={"ID":"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b","Type":"ContainerDied","Data":"71c8d1d862808bd2d3c4269bf38cdfe29ce72941f3b3996e242d27a70442f272"}
Oct 30 03:30:02.367651 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.367628    4278 scope.go:122] "RemoveContainer" containerID="71c8d1d862808bd2d3c4269bf38cdfe29ce72941f3b3996e242d27a70442f272"
Oct 30 03:30:02.369315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.369258    4278 generic.go:356] "Generic (PLEG): container finished" podID="037768be-210c-4930-a922-a1cfd9e8137e" containerID="906629797f2f6c732335c4527e03f0c2ba704b441645d493f9bb8173bbf1a005" exitCode=60
Oct 30 03:30:02.369418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.369395    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" event={"ID":"037768be-210c-4930-a922-a1cfd9e8137e","Type":"ContainerDied","Data":"906629797f2f6c732335c4527e03f0c2ba704b441645d493f9bb8173bbf1a005"}
Oct 30 03:30:02.369650 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.369630    4278 scope.go:122] "RemoveContainer" containerID="906629797f2f6c732335c4527e03f0c2ba704b441645d493f9bb8173bbf1a005"
Oct 30 03:30:02.371260 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.371239    4278 generic.go:356] "Generic (PLEG): container finished" podID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerID="89f956f60276f9ab46673dca2839edc4dc33fdf4a6d4dad99b7aad6f71e61ca6" exitCode=60
Oct 30 03:30:02.371338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.371319    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" event={"ID":"fe31905f-5f69-451b-95c6-dd619b33cb29","Type":"ContainerDied","Data":"89f956f60276f9ab46673dca2839edc4dc33fdf4a6d4dad99b7aad6f71e61ca6"}
Oct 30 03:30:02.371580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.371565    4278 scope.go:122] "RemoveContainer" containerID="89f956f60276f9ab46673dca2839edc4dc33fdf4a6d4dad99b7aad6f71e61ca6"
Oct 30 03:30:02.373211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.373189    4278 generic.go:356] "Generic (PLEG): container finished" podID="212e361c-7a49-4276-be50-34407c4bd146" containerID="c2d5f8163701f59104dcdef5e0c9820efe6856f3e23824030b72f7dbe2d28b82" exitCode=60
Oct 30 03:30:02.373367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.373251    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl" event={"ID":"212e361c-7a49-4276-be50-34407c4bd146","Type":"ContainerDied","Data":"c2d5f8163701f59104dcdef5e0c9820efe6856f3e23824030b72f7dbe2d28b82"}
Oct 30 03:30:02.373491 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.373475    4278 scope.go:122] "RemoveContainer" containerID="c2d5f8163701f59104dcdef5e0c9820efe6856f3e23824030b72f7dbe2d28b82"
Oct 30 03:30:02.379743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.379708    4278 generic.go:356] "Generic (PLEG): container finished" podID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerID="bbe36767b4c2a878787e30d893bc21b2f7aaa6884457500f38d567003eb366cf" exitCode=60
Oct 30 03:30:02.379871 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.379761    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" event={"ID":"5ba040b1-30f4-49c5-9168-a41ddf161956","Type":"ContainerDied","Data":"bbe36767b4c2a878787e30d893bc21b2f7aaa6884457500f38d567003eb366cf"}
Oct 30 03:30:02.380022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:02.380009    4278 scope.go:122] "RemoveContainer" containerID="bbe36767b4c2a878787e30d893bc21b2f7aaa6884457500f38d567003eb366cf"
Oct 30 03:30:03.388891 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:03.388849    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" event={"ID":"fe31905f-5f69-451b-95c6-dd619b33cb29","Type":"ContainerStarted","Data":"d346cb5482214755ba1c241663d7234fea04f9e633f76cad01fb85aca04f6b0e"}
Oct 30 03:30:03.390969 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:03.390944    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl" event={"ID":"212e361c-7a49-4276-be50-34407c4bd146","Type":"ContainerStarted","Data":"5db05c011309b49acadf408654ab0edd46e57afad8c8339cb071bebd917b2b7e"}
Oct 30 03:30:03.392959 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:03.392933    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" event={"ID":"5ba040b1-30f4-49c5-9168-a41ddf161956","Type":"ContainerStarted","Data":"2400e73847ab2520f2bb486f79bdcff003d9f74d05ab0ffbbf8db83b86ee52f8"}
Oct 30 03:30:03.394870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:03.394846    4278 generic.go:356] "Generic (PLEG): container finished" podID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerID="992bee7be635417c724c047f8e2dbdd7a598aaaa32509202734b93faa2a97037" exitCode=60
Oct 30 03:30:03.394989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:03.394921    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq" event={"ID":"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b","Type":"ContainerDied","Data":"992bee7be635417c724c047f8e2dbdd7a598aaaa32509202734b93faa2a97037"}
Oct 30 03:30:03.394989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:03.394975    4278 scope.go:122] "RemoveContainer" containerID="71c8d1d862808bd2d3c4269bf38cdfe29ce72941f3b3996e242d27a70442f272"
Oct 30 03:30:03.395147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:03.395122    4278 scope.go:122] "RemoveContainer" containerID="992bee7be635417c724c047f8e2dbdd7a598aaaa32509202734b93faa2a97037"
Oct 30 03:30:03.395415 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:03.395395    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq_odh-ai-gateway-infra(3baf589a-f6ff-4f1b-8402-e69d9bb0f98b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b"
Oct 30 03:30:03.397043 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:03.397023    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" event={"ID":"037768be-210c-4930-a922-a1cfd9e8137e","Type":"ContainerStarted","Data":"25cf87e23ab513e376ad17f3ae8754702f659c33cf856449997317ae44b5662e"}
Oct 30 03:30:03.538719 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:03.538664    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" podStartSLOduration=3.5386498250000002 podStartE2EDuration="3.538649825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:03.535326603 +0000 UTC m=+2941.459534790" watchObservedRunningTime="2026-10-30 03:30:03.538649825 +0000 UTC m=+2941.462858013"
Oct 30 03:30:04.406911 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.406878    4278 generic.go:356] "Generic (PLEG): container finished" podID="037768be-210c-4930-a922-a1cfd9e8137e" containerID="25cf87e23ab513e376ad17f3ae8754702f659c33cf856449997317ae44b5662e" exitCode=60
Oct 30 03:30:04.407415 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.406939    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" event={"ID":"037768be-210c-4930-a922-a1cfd9e8137e","Type":"ContainerDied","Data":"25cf87e23ab513e376ad17f3ae8754702f659c33cf856449997317ae44b5662e"}
Oct 30 03:30:04.407415 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.407008    4278 scope.go:122] "RemoveContainer" containerID="906629797f2f6c732335c4527e03f0c2ba704b441645d493f9bb8173bbf1a005"
Oct 30 03:30:04.407415 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.407341    4278 scope.go:122] "RemoveContainer" containerID="25cf87e23ab513e376ad17f3ae8754702f659c33cf856449997317ae44b5662e"
Oct 30 03:30:04.407741 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:04.407717    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs_odh-ai-gateway-infra(037768be-210c-4930-a922-a1cfd9e8137e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" podUID="037768be-210c-4930-a922-a1cfd9e8137e"
Oct 30 03:30:04.409346 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.409322    4278 generic.go:356] "Generic (PLEG): container finished" podID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerID="d346cb5482214755ba1c241663d7234fea04f9e633f76cad01fb85aca04f6b0e" exitCode=60
Oct 30 03:30:04.409461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.409408    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" event={"ID":"fe31905f-5f69-451b-95c6-dd619b33cb29","Type":"ContainerDied","Data":"d346cb5482214755ba1c241663d7234fea04f9e633f76cad01fb85aca04f6b0e"}
Oct 30 03:30:04.409665 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.409650    4278 scope.go:122] "RemoveContainer" containerID="d346cb5482214755ba1c241663d7234fea04f9e633f76cad01fb85aca04f6b0e"
Oct 30 03:30:04.409940 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:04.409911    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf_odh-ai-gateway-infra(fe31905f-5f69-451b-95c6-dd619b33cb29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29"
Oct 30 03:30:04.411336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.411311    4278 generic.go:356] "Generic (PLEG): container finished" podID="212e361c-7a49-4276-be50-34407c4bd146" containerID="5db05c011309b49acadf408654ab0edd46e57afad8c8339cb071bebd917b2b7e" exitCode=60
Oct 30 03:30:04.411419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.411397    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl" event={"ID":"212e361c-7a49-4276-be50-34407c4bd146","Type":"ContainerDied","Data":"5db05c011309b49acadf408654ab0edd46e57afad8c8339cb071bebd917b2b7e"}
Oct 30 03:30:04.411689 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.411670    4278 scope.go:122] "RemoveContainer" containerID="5db05c011309b49acadf408654ab0edd46e57afad8c8339cb071bebd917b2b7e"
Oct 30 03:30:04.411954 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:04.411935    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl_odh-ai-gateway-infra(212e361c-7a49-4276-be50-34407c4bd146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl" podUID="212e361c-7a49-4276-be50-34407c4bd146"
Oct 30 03:30:04.413539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.413515    4278 generic.go:356] "Generic (PLEG): container finished" podID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerID="2400e73847ab2520f2bb486f79bdcff003d9f74d05ab0ffbbf8db83b86ee52f8" exitCode=60
Oct 30 03:30:04.413632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.413591    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" event={"ID":"5ba040b1-30f4-49c5-9168-a41ddf161956","Type":"ContainerDied","Data":"2400e73847ab2520f2bb486f79bdcff003d9f74d05ab0ffbbf8db83b86ee52f8"}
Oct 30 03:30:04.413806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.413789    4278 scope.go:122] "RemoveContainer" containerID="2400e73847ab2520f2bb486f79bdcff003d9f74d05ab0ffbbf8db83b86ee52f8"
Oct 30 03:30:04.414108 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:04.414084    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm_odh-ai-gateway-infra(5ba040b1-30f4-49c5-9168-a41ddf161956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956"
Oct 30 03:30:04.415942 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.415925    4278 scope.go:122] "RemoveContainer" containerID="992bee7be635417c724c047f8e2dbdd7a598aaaa32509202734b93faa2a97037"
Oct 30 03:30:04.416143 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:04.416127    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq_odh-ai-gateway-infra(3baf589a-f6ff-4f1b-8402-e69d9bb0f98b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b"
Oct 30 03:30:04.421078 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.421059    4278 scope.go:122] "RemoveContainer" containerID="89f956f60276f9ab46673dca2839edc4dc33fdf4a6d4dad99b7aad6f71e61ca6"
Oct 30 03:30:04.434844 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.434813    4278 scope.go:122] "RemoveContainer" containerID="c2d5f8163701f59104dcdef5e0c9820efe6856f3e23824030b72f7dbe2d28b82"
Oct 30 03:30:04.454246 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:04.454214    4278 scope.go:122] "RemoveContainer" containerID="bbe36767b4c2a878787e30d893bc21b2f7aaa6884457500f38d567003eb366cf"
Oct 30 03:30:05.173730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:05.173690    4278 scope.go:122] "RemoveContainer" containerID="3b6106ef12c0c6eec0d36b8af51e744266b6326482be298443f48d4f9f9fadd0"
Oct 30 03:30:05.184931 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:05.184905    4278 scope.go:122] "RemoveContainer" containerID="dd397696f8ea5a67ce850d38f9947ad00e99dc32e9b65f3103dd2db828386d71"
Oct 30 03:30:05.194952 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:05.194929    4278 scope.go:122] "RemoveContainer" containerID="cd664a5b34ba29ef3a50c114f6ae14cba0d66f78a678dccf093c5f9e666c763f"
Oct 30 03:30:05.204855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:05.204829    4278 scope.go:122] "RemoveContainer" containerID="72c42375f09e20915577d74efa31e4814a398f8f01e737aa49923b92990b3581"
Oct 30 03:30:05.214593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:05.214567    4278 scope.go:122] "RemoveContainer" containerID="2f6ab60a35d24a1d2ab16bb186e05cbce04421b938cc4b2b45375a0c7e447f7d"
Oct 30 03:30:05.424829 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:05.424748    4278 scope.go:122] "RemoveContainer" containerID="25cf87e23ab513e376ad17f3ae8754702f659c33cf856449997317ae44b5662e"
Oct 30 03:30:05.425264 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:05.425048    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs_odh-ai-gateway-infra(037768be-210c-4930-a922-a1cfd9e8137e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" podUID="037768be-210c-4930-a922-a1cfd9e8137e"
Oct 30 03:30:05.426575 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:05.426552    4278 scope.go:122] "RemoveContainer" containerID="d346cb5482214755ba1c241663d7234fea04f9e633f76cad01fb85aca04f6b0e"
Oct 30 03:30:05.426854 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:05.426832    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf_odh-ai-gateway-infra(fe31905f-5f69-451b-95c6-dd619b33cb29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29"
Oct 30 03:30:05.428303 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:05.428270    4278 scope.go:122] "RemoveContainer" containerID="5db05c011309b49acadf408654ab0edd46e57afad8c8339cb071bebd917b2b7e"
Oct 30 03:30:05.428571 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:05.428545    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl_odh-ai-gateway-infra(212e361c-7a49-4276-be50-34407c4bd146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl" podUID="212e361c-7a49-4276-be50-34407c4bd146"
Oct 30 03:30:05.429950 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:05.429932    4278 scope.go:122] "RemoveContainer" containerID="2400e73847ab2520f2bb486f79bdcff003d9f74d05ab0ffbbf8db83b86ee52f8"
Oct 30 03:30:05.430141 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:05.430119    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm_odh-ai-gateway-infra(5ba040b1-30f4-49c5-9168-a41ddf161956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956"
Oct 30 03:30:08.214579 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.214530    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"]
Oct 30 03:30:08.411565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.411532    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"
Oct 30 03:30:08.455496 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.455452    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl" event={"ID":"212e361c-7a49-4276-be50-34407c4bd146","Type":"ContainerDied","Data":"1b3c61abe22dc1f7f594f336acf3365a8c714d9fead67e02bce1a9df32d64d0a"}
Oct 30 03:30:08.455762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.455543    4278 scope.go:122] "RemoveContainer" containerID="5db05c011309b49acadf408654ab0edd46e57afad8c8339cb071bebd917b2b7e"
Oct 30 03:30:08.455762 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.455460    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"
Oct 30 03:30:08.569009 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.568974    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212e361c-7a49-4276-be50-34407c4bd146-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212e361c-7a49-4276-be50-34407c4bd146-maas-api-service-ca\") pod \"212e361c-7a49-4276-be50-34407c4bd146\" (UID: \"212e361c-7a49-4276-be50-34407c4bd146\") "
Oct 30 03:30:08.569234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.569041    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212e361c-7a49-4276-be50-34407c4bd146-kube-api-access-xvklh\" (UniqueName: \"kubernetes.io/projected/212e361c-7a49-4276-be50-34407c4bd146-kube-api-access-xvklh\") pod \"212e361c-7a49-4276-be50-34407c4bd146\" (UID: \"212e361c-7a49-4276-be50-34407c4bd146\") "
Oct 30 03:30:08.569365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.569342    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212e361c-7a49-4276-be50-34407c4bd146-maas-api-service-ca" pod "212e361c-7a49-4276-be50-34407c4bd146" (UID: "212e361c-7a49-4276-be50-34407c4bd146"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:08.569572 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.569552    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212e361c-7a49-4276-be50-34407c4bd146-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:08.571338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.571316    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212e361c-7a49-4276-be50-34407c4bd146-kube-api-access-xvklh" pod "212e361c-7a49-4276-be50-34407c4bd146" (UID: "212e361c-7a49-4276-be50-34407c4bd146"). InnerVolumeSpecName "kube-api-access-xvklh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:08.670794 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.670752    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvklh\" (UniqueName: \"kubernetes.io/projected/212e361c-7a49-4276-be50-34407c4bd146-kube-api-access-xvklh\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:08.800495 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.800452    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"]
Oct 30 03:30:08.811639 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.811611    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888850-lxrhl"]
Oct 30 03:30:08.911367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.911330    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"]
Oct 30 03:30:08.914332 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:08.914302    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ae1dd9-mnkxp"]
Oct 30 03:30:10.198322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:10.198286    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212e361c-7a49-4276-be50-34407c4bd146" path="/var/lib/kubelet/pods/212e361c-7a49-4276-be50-34407c4bd146/volumes"
Oct 30 03:30:10.198774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:10.198758    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ee9891-edd4-4c0e-b066-b52c9ef78e2a" path="/var/lib/kubelet/pods/c0ee9891-edd4-4c0e-b066-b52c9ef78e2a/volumes"
Oct 30 03:30:14.192067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:14.192028    4278 scope.go:122] "RemoveContainer" containerID="2400e73847ab2520f2bb486f79bdcff003d9f74d05ab0ffbbf8db83b86ee52f8"
Oct 30 03:30:14.497147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:14.497109    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" event={"ID":"5ba040b1-30f4-49c5-9168-a41ddf161956","Type":"ContainerStarted","Data":"2765a939fa2f7c8037cd6030104c680fdd7ff96b3644451f4fd25496eabc0d03"}
Oct 30 03:30:15.192907 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:15.192868    4278 scope.go:122] "RemoveContainer" containerID="d346cb5482214755ba1c241663d7234fea04f9e633f76cad01fb85aca04f6b0e"
Oct 30 03:30:15.439171 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:15.439126    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"]
Oct 30 03:30:15.439459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:15.439400    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy" containerID="cri-o://96843cdca12f093c54e2ac43f2774841fe0a97c9a1fd326b8f9401bbd47ba711" gracePeriod=30
Oct 30 03:30:15.485365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:15.485287    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:30:15.506060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:15.506028    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" event={"ID":"fe31905f-5f69-451b-95c6-dd619b33cb29","Type":"ContainerStarted","Data":"6e655c99a54685c12e24cf8b3ef1472132eabcdd92a4cf92522bcad1441b476c"}
Oct 30 03:30:15.508060 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:15.508032    4278 generic.go:356] "Generic (PLEG): container finished" podID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerID="2765a939fa2f7c8037cd6030104c680fdd7ff96b3644451f4fd25496eabc0d03" exitCode=60
Oct 30 03:30:15.508208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:15.508104    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" event={"ID":"5ba040b1-30f4-49c5-9168-a41ddf161956","Type":"ContainerDied","Data":"2765a939fa2f7c8037cd6030104c680fdd7ff96b3644451f4fd25496eabc0d03"}
Oct 30 03:30:15.508208 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:15.508156    4278 scope.go:122] "RemoveContainer" containerID="2400e73847ab2520f2bb486f79bdcff003d9f74d05ab0ffbbf8db83b86ee52f8"
Oct 30 03:30:15.508543 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:15.508522    4278 scope.go:122] "RemoveContainer" containerID="2765a939fa2f7c8037cd6030104c680fdd7ff96b3644451f4fd25496eabc0d03"
Oct 30 03:30:15.508826 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:15.508805    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm_odh-ai-gateway-infra(5ba040b1-30f4-49c5-9168-a41ddf161956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956"
Oct 30 03:30:15.555776 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:15.555733    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"]
Oct 30 03:30:16.136697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.136654    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"]
Oct 30 03:30:16.137174 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.137159    4278 cpu_manager.go:414] "containerMap: removing container" podUID="212e361c-7a49-4276-be50-34407c4bd146" containerName="cleanup"
Oct 30 03:30:16.137234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.137175    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212e361c-7a49-4276-be50-34407c4bd146" containerName="cleanup"
Oct 30 03:30:16.137234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.137188    4278 cpu_manager.go:414] "containerMap: removing container" podUID="212e361c-7a49-4276-be50-34407c4bd146" containerName="cleanup"
Oct 30 03:30:16.137234 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.137195    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212e361c-7a49-4276-be50-34407c4bd146" containerName="cleanup"
Oct 30 03:30:16.137344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.137305    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="212e361c-7a49-4276-be50-34407c4bd146" containerName="cleanup"
Oct 30 03:30:16.137344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.137314    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="212e361c-7a49-4276-be50-34407c4bd146" containerName="cleanup"
Oct 30 03:30:16.140700 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.140681    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.160352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.160320    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"]
Oct 30 03:30:16.173800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.173772    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:30:16.192193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.192161    4278 scope.go:122] "RemoveContainer" containerID="25cf87e23ab513e376ad17f3ae8754702f659c33cf856449997317ae44b5662e"
Oct 30 03:30:16.268765 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.268733    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.269114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.268782    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.269114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.268905    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ddc\" (UniqueName: \"kubernetes.io/projected/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-kube-api-access-h9ddc\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.369822 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.369785    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.369822 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.369831    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.370125 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.369958    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ddc\" (UniqueName: \"kubernetes.io/projected/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-kube-api-access-h9ddc\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.370542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.370515    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.372291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.372264    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.391242 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.391206    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ddc\" (UniqueName: \"kubernetes.io/projected/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-kube-api-access-h9ddc\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.454653 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.454616    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:16.520574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.519942    4278 generic.go:356] "Generic (PLEG): container finished" podID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerID="6e655c99a54685c12e24cf8b3ef1472132eabcdd92a4cf92522bcad1441b476c" exitCode=60
Oct 30 03:30:16.520574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.520030    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" event={"ID":"fe31905f-5f69-451b-95c6-dd619b33cb29","Type":"ContainerDied","Data":"6e655c99a54685c12e24cf8b3ef1472132eabcdd92a4cf92522bcad1441b476c"}
Oct 30 03:30:16.520574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.520086    4278 scope.go:122] "RemoveContainer" containerID="d346cb5482214755ba1c241663d7234fea04f9e633f76cad01fb85aca04f6b0e"
Oct 30 03:30:16.520574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.520526    4278 scope.go:122] "RemoveContainer" containerID="6e655c99a54685c12e24cf8b3ef1472132eabcdd92a4cf92522bcad1441b476c"
Oct 30 03:30:16.521212 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:16.520886    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf_odh-ai-gateway-infra(fe31905f-5f69-451b-95c6-dd619b33cb29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29"
Oct 30 03:30:16.530152 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.530101    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" event={"ID":"037768be-210c-4930-a922-a1cfd9e8137e","Type":"ContainerStarted","Data":"f8c9c5a3f43128a196568ffdfa01f84e7b5ca417dad8e23289897b94ceeec52b"}
Oct 30 03:30:16.574869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.574831    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"]
Oct 30 03:30:16.674344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.674314    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"
Oct 30 03:30:16.725085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.725045    4278 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:30:16.725244 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.725113    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:30:16.756451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.756396    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"]
Oct 30 03:30:16.760265 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:30:16.760197    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0cbd8d_88d9_4edf_9131_ce6627a8f2fa.slice/crio-2b84b590123cd36bcab259888f2641221ffb31345a5e37bad21375775187c251 WatchSource:0}: Error finding container 2b84b590123cd36bcab259888f2641221ffb31345a5e37bad21375775187c251: Status 404 returned error can't find the container with id 2b84b590123cd36bcab259888f2641221ffb31345a5e37bad21375775187c251
Oct 30 03:30:16.775488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.775458    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba040b1-30f4-49c5-9168-a41ddf161956-kube-api-access-b88qz\" (UniqueName: \"kubernetes.io/projected/5ba040b1-30f4-49c5-9168-a41ddf161956-kube-api-access-b88qz\") pod \"5ba040b1-30f4-49c5-9168-a41ddf161956\" (UID: \"5ba040b1-30f4-49c5-9168-a41ddf161956\") "
Oct 30 03:30:16.775615 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.775523    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba040b1-30f4-49c5-9168-a41ddf161956-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba040b1-30f4-49c5-9168-a41ddf161956-maas-api-service-ca\") pod \"5ba040b1-30f4-49c5-9168-a41ddf161956\" (UID: \"5ba040b1-30f4-49c5-9168-a41ddf161956\") "
Oct 30 03:30:16.775872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.775852    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba040b1-30f4-49c5-9168-a41ddf161956-maas-api-service-ca" pod "5ba040b1-30f4-49c5-9168-a41ddf161956" (UID: "5ba040b1-30f4-49c5-9168-a41ddf161956"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:16.777699 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.777672    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba040b1-30f4-49c5-9168-a41ddf161956-kube-api-access-b88qz" pod "5ba040b1-30f4-49c5-9168-a41ddf161956" (UID: "5ba040b1-30f4-49c5-9168-a41ddf161956"). InnerVolumeSpecName "kube-api-access-b88qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:16.876305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.876276    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88qz\" (UniqueName: \"kubernetes.io/projected/5ba040b1-30f4-49c5-9168-a41ddf161956-kube-api-access-b88qz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:16.876305 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:16.876301    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba040b1-30f4-49c5-9168-a41ddf161956-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:17.359245 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.359146    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"]
Oct 30 03:30:17.359951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.359929    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerName="cleanup"
Oct 30 03:30:17.359997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.359954    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerName="cleanup"
Oct 30 03:30:17.359997 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.359994    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerName="cleanup"
Oct 30 03:30:17.360059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.360002    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerName="cleanup"
Oct 30 03:30:17.360155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.360144    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerName="cleanup"
Oct 30 03:30:17.360189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.360161    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerName="cleanup"
Oct 30 03:30:17.360189 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.360172    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerName="cleanup"
Oct 30 03:30:17.363820 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.363795    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.381016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.380978    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-203899d4-k9lw5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.381247 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.381198    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nbgz\" (UniqueName: \"kubernetes.io/projected/5b20a143-dfe1-479a-9f5e-04e84278c4f5-kube-api-access-8nbgz\") pod \"maas-api-revoke-sub-203899d4-k9lw5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.381340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.381275    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-203899d4-k9lw5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.382134 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.382109    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"]
Oct 30 03:30:17.482384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.482344    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-203899d4-k9lw5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.482639 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.482501    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nbgz\" (UniqueName: \"kubernetes.io/projected/5b20a143-dfe1-479a-9f5e-04e84278c4f5-kube-api-access-8nbgz\") pod \"maas-api-revoke-sub-203899d4-k9lw5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.482639 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.482558    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-203899d4-k9lw5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.483143 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.483119    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-203899d4-k9lw5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.485353 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.485329    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-203899d4-k9lw5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.501603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.501573    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nbgz\" (UniqueName: \"kubernetes.io/projected/5b20a143-dfe1-479a-9f5e-04e84278c4f5-kube-api-access-8nbgz\") pod \"maas-api-revoke-sub-203899d4-k9lw5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.540000 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.539972    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"
Oct 30 03:30:17.540154 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.540003    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm" event={"ID":"5ba040b1-30f4-49c5-9168-a41ddf161956","Type":"ContainerDied","Data":"ae25b8f5613af05f063ab2f58a556d2bf13e058b460a382d3816a1d0534c23a6"}
Oct 30 03:30:17.540154 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.540080    4278 scope.go:122] "RemoveContainer" containerID="2765a939fa2f7c8037cd6030104c680fdd7ff96b3644451f4fd25496eabc0d03"
Oct 30 03:30:17.542232 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.542202    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7" event={"ID":"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa","Type":"ContainerStarted","Data":"a694b93f458413ae844a18919302f127966562411330a48301ba8d0257d3ea7e"}
Oct 30 03:30:17.542371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.542259    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7" event={"ID":"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa","Type":"ContainerStarted","Data":"2b84b590123cd36bcab259888f2641221ffb31345a5e37bad21375775187c251"}
Oct 30 03:30:17.544777 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.544746    4278 generic.go:356] "Generic (PLEG): container finished" podID="037768be-210c-4930-a922-a1cfd9e8137e" containerID="f8c9c5a3f43128a196568ffdfa01f84e7b5ca417dad8e23289897b94ceeec52b" exitCode=60
Oct 30 03:30:17.544874 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.544787    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" event={"ID":"037768be-210c-4930-a922-a1cfd9e8137e","Type":"ContainerDied","Data":"f8c9c5a3f43128a196568ffdfa01f84e7b5ca417dad8e23289897b94ceeec52b"}
Oct 30 03:30:17.545193 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.545174    4278 scope.go:122] "RemoveContainer" containerID="f8c9c5a3f43128a196568ffdfa01f84e7b5ca417dad8e23289897b94ceeec52b"
Oct 30 03:30:17.545528 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:17.545501    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs_odh-ai-gateway-infra(037768be-210c-4930-a922-a1cfd9e8137e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" podUID="037768be-210c-4930-a922-a1cfd9e8137e"
Oct 30 03:30:17.554016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.553875    4278 scope.go:122] "RemoveContainer" containerID="25cf87e23ab513e376ad17f3ae8754702f659c33cf856449997317ae44b5662e"
Oct 30 03:30:17.620263 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.620179    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"]
Oct 30 03:30:17.632661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.627144    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-aeff1d-29888850-gdqnm"]
Oct 30 03:30:17.650056 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.650021    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"]
Oct 30 03:30:17.677808 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.677779    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:17.700322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.700256    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7" podStartSLOduration=1.7002237679999999 podStartE2EDuration="1.700223768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:17.699330016 +0000 UTC m=+2955.623538205" watchObservedRunningTime="2026-10-30 03:30:17.700223768 +0000 UTC m=+2955.624431958"
Oct 30 03:30:17.785226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.785200    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"
Oct 30 03:30:17.785639 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.785615    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe31905f-5f69-451b-95c6-dd619b33cb29-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe31905f-5f69-451b-95c6-dd619b33cb29-maas-api-service-ca\") pod \"fe31905f-5f69-451b-95c6-dd619b33cb29\" (UID: \"fe31905f-5f69-451b-95c6-dd619b33cb29\") "
Oct 30 03:30:17.785700 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.785687    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe31905f-5f69-451b-95c6-dd619b33cb29-kube-api-access-wqw9c\" (UniqueName: \"kubernetes.io/projected/fe31905f-5f69-451b-95c6-dd619b33cb29-kube-api-access-wqw9c\") pod \"fe31905f-5f69-451b-95c6-dd619b33cb29\" (UID: \"fe31905f-5f69-451b-95c6-dd619b33cb29\") "
Oct 30 03:30:17.786620 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.786589    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe31905f-5f69-451b-95c6-dd619b33cb29-maas-api-service-ca" pod "fe31905f-5f69-451b-95c6-dd619b33cb29" (UID: "fe31905f-5f69-451b-95c6-dd619b33cb29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:17.788592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.788561    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe31905f-5f69-451b-95c6-dd619b33cb29-kube-api-access-wqw9c" pod "fe31905f-5f69-451b-95c6-dd619b33cb29" (UID: "fe31905f-5f69-451b-95c6-dd619b33cb29"). InnerVolumeSpecName "kube-api-access-wqw9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:17.887204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.887118    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe31905f-5f69-451b-95c6-dd619b33cb29-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:17.887204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.887150    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqw9c\" (UniqueName: \"kubernetes.io/projected/fe31905f-5f69-451b-95c6-dd619b33cb29-kube-api-access-wqw9c\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:17.992015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:17.991892    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"]
Oct 30 03:30:17.994661 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:30:17.994627    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b20a143_dfe1_479a_9f5e_04e84278c4f5.slice/crio-890024217e6b4a5112051040867bbe485bb7984668d1ea662c0ebe98c5d65263 WatchSource:0}: Error finding container 890024217e6b4a5112051040867bbe485bb7984668d1ea662c0ebe98c5d65263: Status 404 returned error can't find the container with id 890024217e6b4a5112051040867bbe485bb7984668d1ea662c0ebe98c5d65263
Oct 30 03:30:18.191880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.191847    4278 scope.go:122] "RemoveContainer" containerID="992bee7be635417c724c047f8e2dbdd7a598aaaa32509202734b93faa2a97037"
Oct 30 03:30:18.198960 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.198933    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956" path="/var/lib/kubelet/pods/5ba040b1-30f4-49c5-9168-a41ddf161956/volumes"
Oct 30 03:30:18.554255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.554217    4278 generic.go:356] "Generic (PLEG): container finished" podID="be0cbd8d-88d9-4edf-9131-ce6627a8f2fa" containerID="a694b93f458413ae844a18919302f127966562411330a48301ba8d0257d3ea7e" exitCode=0
Oct 30 03:30:18.554735 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.554298    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7" event={"ID":"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa","Type":"ContainerDied","Data":"a694b93f458413ae844a18919302f127966562411330a48301ba8d0257d3ea7e"}
Oct 30 03:30:18.556747 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.556717    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq" event={"ID":"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b","Type":"ContainerStarted","Data":"abf148d86244b49589e18d799ddbaa9cbcf6ac51c104f04d191af459fd88425b"}
Oct 30 03:30:18.558488 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.558459    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5" event={"ID":"5b20a143-dfe1-479a-9f5e-04e84278c4f5","Type":"ContainerStarted","Data":"387e3f2e8c44689e60031354c872b5fef84fa9900c48338e617d7f17712e6307"}
Oct 30 03:30:18.558657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.558509    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5" event={"ID":"5b20a143-dfe1-479a-9f5e-04e84278c4f5","Type":"ContainerStarted","Data":"890024217e6b4a5112051040867bbe485bb7984668d1ea662c0ebe98c5d65263"}
Oct 30 03:30:18.563134 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.563113    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"
Oct 30 03:30:18.563263 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.563141    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf" event={"ID":"fe31905f-5f69-451b-95c6-dd619b33cb29","Type":"ContainerDied","Data":"d8a7a59dda182692323804b602a17b845ac7e21f4addf88a91436326ece3a72e"}
Oct 30 03:30:18.563263 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.563207    4278 scope.go:122] "RemoveContainer" containerID="6e655c99a54685c12e24cf8b3ef1472132eabcdd92a4cf92522bcad1441b476c"
Oct 30 03:30:18.637805 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.637735    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5" podStartSLOduration=1.637711163 podStartE2EDuration="1.637711163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:18.630301553 +0000 UTC m=+2956.554509741" watchObservedRunningTime="2026-10-30 03:30:18.637711163 +0000 UTC m=+2956.561919352"
Oct 30 03:30:18.668074 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.668009    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq" podStartSLOduration=18.667990525 podStartE2EDuration="18.667990525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:18.666584064 +0000 UTC m=+2956.590792252" watchObservedRunningTime="2026-10-30 03:30:18.667990525 +0000 UTC m=+2956.592198714"
Oct 30 03:30:18.715080 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.715055    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"
Oct 30 03:30:18.737936 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.737897    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"]
Oct 30 03:30:18.740873 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.740843    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cf605a-29888850-ztvbf"]
Oct 30 03:30:18.795802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.795763    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037768be-210c-4930-a922-a1cfd9e8137e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037768be-210c-4930-a922-a1cfd9e8137e-maas-api-service-ca\") pod \"037768be-210c-4930-a922-a1cfd9e8137e\" (UID: \"037768be-210c-4930-a922-a1cfd9e8137e\") "
Oct 30 03:30:18.796022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.795957    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037768be-210c-4930-a922-a1cfd9e8137e-kube-api-access-c5k67\" (UniqueName: \"kubernetes.io/projected/037768be-210c-4930-a922-a1cfd9e8137e-kube-api-access-c5k67\") pod \"037768be-210c-4930-a922-a1cfd9e8137e\" (UID: \"037768be-210c-4930-a922-a1cfd9e8137e\") "
Oct 30 03:30:18.796156 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.796131    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037768be-210c-4930-a922-a1cfd9e8137e-maas-api-service-ca" pod "037768be-210c-4930-a922-a1cfd9e8137e" (UID: "037768be-210c-4930-a922-a1cfd9e8137e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:18.796387 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.796366    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037768be-210c-4930-a922-a1cfd9e8137e-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:18.798243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.798214    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037768be-210c-4930-a922-a1cfd9e8137e-kube-api-access-c5k67" pod "037768be-210c-4930-a922-a1cfd9e8137e" (UID: "037768be-210c-4930-a922-a1cfd9e8137e"). InnerVolumeSpecName "kube-api-access-c5k67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:18.898081 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:18.898043    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5k67\" (UniqueName: \"kubernetes.io/projected/037768be-210c-4930-a922-a1cfd9e8137e-kube-api-access-c5k67\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:19.573779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.573737    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"
Oct 30 03:30:19.573779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.573762    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs" event={"ID":"037768be-210c-4930-a922-a1cfd9e8137e","Type":"ContainerDied","Data":"bee3f04cc44b4ee0f2f4540bcb8d91ceb74a430a2335c20e197998ab126c1967"}
Oct 30 03:30:19.574321 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.573838    4278 scope.go:122] "RemoveContainer" containerID="f8c9c5a3f43128a196568ffdfa01f84e7b5ca417dad8e23289897b94ceeec52b"
Oct 30 03:30:19.578598 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.578567    4278 generic.go:356] "Generic (PLEG): container finished" podID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerID="abf148d86244b49589e18d799ddbaa9cbcf6ac51c104f04d191af459fd88425b" exitCode=60
Oct 30 03:30:19.578725 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.578631    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq" event={"ID":"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b","Type":"ContainerDied","Data":"abf148d86244b49589e18d799ddbaa9cbcf6ac51c104f04d191af459fd88425b"}
Oct 30 03:30:19.579271 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.579087    4278 scope.go:122] "RemoveContainer" containerID="abf148d86244b49589e18d799ddbaa9cbcf6ac51c104f04d191af459fd88425b"
Oct 30 03:30:19.579487 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:19.579453    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq_odh-ai-gateway-infra(3baf589a-f6ff-4f1b-8402-e69d9bb0f98b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b"
Oct 30 03:30:19.580492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.580466    4278 generic.go:356] "Generic (PLEG): container finished" podID="5b20a143-dfe1-479a-9f5e-04e84278c4f5" containerID="387e3f2e8c44689e60031354c872b5fef84fa9900c48338e617d7f17712e6307" exitCode=0
Oct 30 03:30:19.580759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.580736    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5" event={"ID":"5b20a143-dfe1-479a-9f5e-04e84278c4f5","Type":"ContainerDied","Data":"387e3f2e8c44689e60031354c872b5fef84fa9900c48338e617d7f17712e6307"}
Oct 30 03:30:19.586188 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.586170    4278 scope.go:122] "RemoveContainer" containerID="992bee7be635417c724c047f8e2dbdd7a598aaaa32509202734b93faa2a97037"
Oct 30 03:30:19.661388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.661322    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"]
Oct 30 03:30:19.666378 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.666343    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888850-b4wcs"]
Oct 30 03:30:19.678222 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.678187    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"]
Oct 30 03:30:19.736319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.736294    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:19.807858 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.807813    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-kube-api-access-h9ddc\" (UniqueName: \"kubernetes.io/projected/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-kube-api-access-h9ddc\") pod \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") "
Oct 30 03:30:19.808047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.807938    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-xks-ca\") pod \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") "
Oct 30 03:30:19.808132 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.808079    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-openshift-ca\") pod \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\" (UID: \"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa\") "
Oct 30 03:30:19.808567 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.808535    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-openshift-ca" pod "be0cbd8d-88d9-4edf-9131-ce6627a8f2fa" (UID: "be0cbd8d-88d9-4edf-9131-ce6627a8f2fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:19.810172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.810139    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-kube-api-access-h9ddc" pod "be0cbd8d-88d9-4edf-9131-ce6627a8f2fa" (UID: "be0cbd8d-88d9-4edf-9131-ce6627a8f2fa"). InnerVolumeSpecName "kube-api-access-h9ddc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:19.810278 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.810182    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-xks-ca" pod "be0cbd8d-88d9-4edf-9131-ce6627a8f2fa" (UID: "be0cbd8d-88d9-4edf-9131-ce6627a8f2fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:19.909179 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.909086    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9ddc\" (UniqueName: \"kubernetes.io/projected/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-kube-api-access-h9ddc\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:19.909179 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.909122    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:19.909179 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:19.909133    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.199516 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.199374    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037768be-210c-4930-a922-a1cfd9e8137e" path="/var/lib/kubelet/pods/037768be-210c-4930-a922-a1cfd9e8137e/volumes"
Oct 30 03:30:20.200071 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.200051    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29" path="/var/lib/kubelet/pods/fe31905f-5f69-451b-95c6-dd619b33cb29/volumes"
Oct 30 03:30:20.591489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.591453    4278 generic.go:356] "Generic (PLEG): container finished" podID="a7219839-2d46-4a53-9a67-14a21f585686" containerID="96843cdca12f093c54e2ac43f2774841fe0a97c9a1fd326b8f9401bbd47ba711" exitCode=0
Oct 30 03:30:20.592075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.591673    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" event={"ID":"a7219839-2d46-4a53-9a67-14a21f585686","Type":"ContainerDied","Data":"96843cdca12f093c54e2ac43f2774841fe0a97c9a1fd326b8f9401bbd47ba711"}
Oct 30 03:30:20.592075 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.591747    4278 scope.go:122] "RemoveContainer" containerID="ad14d940efda53dfd9e2e444ff77a25676c72ae7325a596432a3ef643238ee7a"
Oct 30 03:30:20.594091 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.594072    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"
Oct 30 03:30:20.594225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.594061    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7" event={"ID":"be0cbd8d-88d9-4edf-9131-ce6627a8f2fa","Type":"ContainerDied","Data":"2b84b590123cd36bcab259888f2641221ffb31345a5e37bad21375775187c251"}
Oct 30 03:30:20.594225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.594200    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b84b590123cd36bcab259888f2641221ffb31345a5e37bad21375775187c251"
Oct 30 03:30:20.727340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.727271    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:30:20.783732 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.783704    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"
Oct 30 03:30:20.789360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.789335    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:20.817919 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.817889    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-envoy\") pod \"a7219839-2d46-4a53-9a67-14a21f585686\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") "
Oct 30 03:30:20.818119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.817941    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-kube-api-access-c8v8x\" (UniqueName: \"kubernetes.io/projected/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-kube-api-access-c8v8x\") pod \"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b\" (UID: \"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b\") "
Oct 30 03:30:20.818119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.817981    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-istio-token\" (UniqueName: \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-istio-token\") pod \"a7219839-2d46-4a53-9a67-14a21f585686\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") "
Oct 30 03:30:20.818119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818031    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-openshift-ca\") pod \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") "
Oct 30 03:30:20.818119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818063    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b20a143-dfe1-479a-9f5e-04e84278c4f5-kube-api-access-8nbgz\" (UniqueName: \"kubernetes.io/projected/5b20a143-dfe1-479a-9f5e-04e84278c4f5-kube-api-access-8nbgz\") pod \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") "
Oct 30 03:30:20.818119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818097    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-credential-socket\") pod \"a7219839-2d46-4a53-9a67-14a21f585686\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") "
Oct 30 03:30:20.818388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818128    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-kube-api-access-n2whl\" (UniqueName: \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-kube-api-access-n2whl\") pod \"a7219839-2d46-4a53-9a67-14a21f585686\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") "
Oct 30 03:30:20.818388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818172    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-data\") pod \"a7219839-2d46-4a53-9a67-14a21f585686\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") "
Oct 30 03:30:20.818388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818243    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-maas-api-service-ca\") pod \"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b\" (UID: \"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b\") "
Oct 30 03:30:20.818388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818330    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-certs\") pod \"a7219839-2d46-4a53-9a67-14a21f585686\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") "
Oct 30 03:30:20.818388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818363    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-socket\") pod \"a7219839-2d46-4a53-9a67-14a21f585686\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") "
Oct 30 03:30:20.818664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818414    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-xks-ca\") pod \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\" (UID: \"5b20a143-dfe1-479a-9f5e-04e84278c4f5\") "
Oct 30 03:30:20.818664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818418    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-openshift-ca" pod "5b20a143-dfe1-479a-9f5e-04e84278c4f5" (UID: "5b20a143-dfe1-479a-9f5e-04e84278c4f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:20.818664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818545    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7219839-2d46-4a53-9a67-14a21f585686-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7219839-2d46-4a53-9a67-14a21f585686-istio-podinfo\") pod \"a7219839-2d46-4a53-9a67-14a21f585686\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") "
Oct 30 03:30:20.818664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818578    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7219839-2d46-4a53-9a67-14a21f585686-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7219839-2d46-4a53-9a67-14a21f585686-istiod-ca-cert\") pod \"a7219839-2d46-4a53-9a67-14a21f585686\" (UID: \"a7219839-2d46-4a53-9a67-14a21f585686\") "
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818676    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-certs" pod "a7219839-2d46-4a53-9a67-14a21f585686" (UID: "a7219839-2d46-4a53-9a67-14a21f585686"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818707    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-data" pod "a7219839-2d46-4a53-9a67-14a21f585686" (UID: "a7219839-2d46-4a53-9a67-14a21f585686"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.818826    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-credential-socket" pod "a7219839-2d46-4a53-9a67-14a21f585686" (UID: "a7219839-2d46-4a53-9a67-14a21f585686"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.819003    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-maas-api-service-ca" pod "3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" (UID: "3baf589a-f6ff-4f1b-8402-e69d9bb0f98b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.819016    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7219839-2d46-4a53-9a67-14a21f585686-istiod-ca-cert" pod "a7219839-2d46-4a53-9a67-14a21f585686" (UID: "a7219839-2d46-4a53-9a67-14a21f585686"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.819089    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.819103    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.819118    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.819129    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.819148    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.819593 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.819467    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-socket" pod "a7219839-2d46-4a53-9a67-14a21f585686" (UID: "a7219839-2d46-4a53-9a67-14a21f585686"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:20.820839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.820810    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-istio-token" pod "a7219839-2d46-4a53-9a67-14a21f585686" (UID: "a7219839-2d46-4a53-9a67-14a21f585686"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:20.820963 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.820854    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-kube-api-access-c8v8x" pod "3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" (UID: "3baf589a-f6ff-4f1b-8402-e69d9bb0f98b"). InnerVolumeSpecName "kube-api-access-c8v8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:20.821217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.821189    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-kube-api-access-n2whl" pod "a7219839-2d46-4a53-9a67-14a21f585686" (UID: "a7219839-2d46-4a53-9a67-14a21f585686"). InnerVolumeSpecName "kube-api-access-n2whl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:20.821343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.821289    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7219839-2d46-4a53-9a67-14a21f585686-istio-podinfo" pod "a7219839-2d46-4a53-9a67-14a21f585686" (UID: "a7219839-2d46-4a53-9a67-14a21f585686"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:30:20.821671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.821649    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b20a143-dfe1-479a-9f5e-04e84278c4f5-kube-api-access-8nbgz" pod "5b20a143-dfe1-479a-9f5e-04e84278c4f5" (UID: "5b20a143-dfe1-479a-9f5e-04e84278c4f5"). InnerVolumeSpecName "kube-api-access-8nbgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:20.821862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.821839    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-xks-ca" pod "5b20a143-dfe1-479a-9f5e-04e84278c4f5" (UID: "5b20a143-dfe1-479a-9f5e-04e84278c4f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:20.821954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.821843    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-envoy" pod "a7219839-2d46-4a53-9a67-14a21f585686" (UID: "a7219839-2d46-4a53-9a67-14a21f585686"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:20.920497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.920457    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.920497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.920494    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b20a143-dfe1-479a-9f5e-04e84278c4f5-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.920497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.920504    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7219839-2d46-4a53-9a67-14a21f585686-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.920774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.920515    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7219839-2d46-4a53-9a67-14a21f585686-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.920774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.920526    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7219839-2d46-4a53-9a67-14a21f585686-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.920774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.920535    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8v8x\" (UniqueName: \"kubernetes.io/projected/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b-kube-api-access-c8v8x\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.920774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.920544    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.920774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.920552    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nbgz\" (UniqueName: \"kubernetes.io/projected/5b20a143-dfe1-479a-9f5e-04e84278c4f5-kube-api-access-8nbgz\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.920774 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.920562    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2whl\" (UniqueName: \"kubernetes.io/projected/a7219839-2d46-4a53-9a67-14a21f585686-kube-api-access-n2whl\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.958810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:20.958772    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"]
Oct 30 03:30:21.605881 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.605852    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"
Oct 30 03:30:21.606354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.605849    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq" event={"ID":"3baf589a-f6ff-4f1b-8402-e69d9bb0f98b","Type":"ContainerDied","Data":"93296da079193ff158e16ad2c493f5784c890a49705d21873e15eeb5a037d81c"}
Oct 30 03:30:21.606354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.606026    4278 scope.go:122] "RemoveContainer" containerID="abf148d86244b49589e18d799ddbaa9cbcf6ac51c104f04d191af459fd88425b"
Oct 30 03:30:21.607559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.607538    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"
Oct 30 03:30:21.607688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.607549    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5" event={"ID":"5b20a143-dfe1-479a-9f5e-04e84278c4f5","Type":"ContainerDied","Data":"890024217e6b4a5112051040867bbe485bb7984668d1ea662c0ebe98c5d65263"}
Oct 30 03:30:21.607688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.607603    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="890024217e6b4a5112051040867bbe485bb7984668d1ea662c0ebe98c5d65263"
Oct 30 03:30:21.609375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.609351    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb" event={"ID":"a7219839-2d46-4a53-9a67-14a21f585686","Type":"ContainerDied","Data":"793622c069c908ddd43cafeb6de74e107c3bd31bdd6de266315d0a46ac3df6be"}
Oct 30 03:30:21.609530 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.609391    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"
Oct 30 03:30:21.617747 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.617689    4278 scope.go:122] "RemoveContainer" containerID="96843cdca12f093c54e2ac43f2774841fe0a97c9a1fd326b8f9401bbd47ba711"
Oct 30 03:30:21.684679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.684645    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"]
Oct 30 03:30:21.688032 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.687999    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3f72d5-29888850-p94fq"]
Oct 30 03:30:21.720344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.720258    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"]
Oct 30 03:30:21.731224 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:21.731187    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-fvxzb"]
Oct 30 03:30:22.199049 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:22.199012    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" path="/var/lib/kubelet/pods/3baf589a-f6ff-4f1b-8402-e69d9bb0f98b/volumes"
Oct 30 03:30:22.199559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:22.199542    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7219839-2d46-4a53-9a67-14a21f585686" path="/var/lib/kubelet/pods/a7219839-2d46-4a53-9a67-14a21f585686/volumes"
Oct 30 03:30:23.027158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:23.027116    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"]
Oct 30 03:30:23.038469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:23.038404    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"]
Oct 30 03:30:23.041361 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:23.041332    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-486f9dbb-gwcxq"]
Oct 30 03:30:23.043675 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:23.043645    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1dff0f-86r9j"]
Oct 30 03:30:24.190444 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:24.190397    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:30:24.198872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:24.198842    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7231391f-da96-46b1-ac3a-ee0f61fe1d3f" path="/var/lib/kubelet/pods/7231391f-da96-46b1-ac3a-ee0f61fe1d3f/volumes"
Oct 30 03:30:24.199336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:24.199321    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37bc90c-c9c6-4983-b3d6-5ca30cd1f467" path="/var/lib/kubelet/pods/e37bc90c-c9c6-4983-b3d6-5ca30cd1f467/volumes"
Oct 30 03:30:30.479592 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.479550    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:30:30.516146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516110    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"]
Oct 30 03:30:30.516768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516743    4278 cpu_manager.go:414] "containerMap: removing container" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerName="cleanup"
Oct 30 03:30:30.516768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516769    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516781    4278 cpu_manager.go:414] "containerMap: removing container" podUID="be0cbd8d-88d9-4edf-9131-ce6627a8f2fa" containerName="revoke-keys"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516787    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0cbd8d-88d9-4edf-9131-ce6627a8f2fa" containerName="revoke-keys"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516796    4278 cpu_manager.go:414] "containerMap: removing container" podUID="037768be-210c-4930-a922-a1cfd9e8137e" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516801    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037768be-210c-4930-a922-a1cfd9e8137e" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516814    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5b20a143-dfe1-479a-9f5e-04e84278c4f5" containerName="revoke-keys"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516821    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b20a143-dfe1-479a-9f5e-04e84278c4f5" containerName="revoke-keys"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516838    4278 cpu_manager.go:414] "containerMap: removing container" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516844    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516854    4278 cpu_manager.go:414] "containerMap: removing container" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516859    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516873    4278 cpu_manager.go:414] "containerMap: removing container" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516878    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516895    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516899    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516910    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516915    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba040b1-30f4-49c5-9168-a41ddf161956" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516921    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516925    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516933    4278 cpu_manager.go:414] "containerMap: removing container" podUID="037768be-210c-4930-a922-a1cfd9e8137e" containerName="cleanup"
Oct 30 03:30:30.516951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.516937    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037768be-210c-4930-a922-a1cfd9e8137e" containerName="cleanup"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517004    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerName="cleanup"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517014    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerName="cleanup"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517021    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerName="cleanup"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517028    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517035    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="037768be-210c-4930-a922-a1cfd9e8137e" containerName="cleanup"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517041    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7219839-2d46-4a53-9a67-14a21f585686" containerName="istio-proxy"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517047    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="037768be-210c-4930-a922-a1cfd9e8137e" containerName="cleanup"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517052    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0cbd8d-88d9-4edf-9131-ce6627a8f2fa" containerName="revoke-keys"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517057    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="037768be-210c-4930-a922-a1cfd9e8137e" containerName="cleanup"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517063    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b20a143-dfe1-479a-9f5e-04e84278c4f5" containerName="revoke-keys"
Oct 30 03:30:30.517652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.517069    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerName="cleanup"
Oct 30 03:30:30.522219 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.522198    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.526158 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.526132    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"]
Oct 30 03:30:30.526584 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.526563    4278 cpu_manager.go:414] "containerMap: removing container" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerName="cleanup"
Oct 30 03:30:30.526584 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.526581    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerName="cleanup"
Oct 30 03:30:30.526787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.526598    4278 cpu_manager.go:414] "containerMap: removing container" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerName="cleanup"
Oct 30 03:30:30.526787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.526604    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerName="cleanup"
Oct 30 03:30:30.526787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.526631    4278 cpu_manager.go:414] "containerMap: removing container" podUID="037768be-210c-4930-a922-a1cfd9e8137e" containerName="cleanup"
Oct 30 03:30:30.526787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.526638    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037768be-210c-4930-a922-a1cfd9e8137e" containerName="cleanup"
Oct 30 03:30:30.526787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.526716    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe31905f-5f69-451b-95c6-dd619b33cb29" containerName="cleanup"
Oct 30 03:30:30.526787 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.526732    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="3baf589a-f6ff-4f1b-8402-e69d9bb0f98b" containerName="cleanup"
Oct 30 03:30:30.529949 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.529928    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:30.538181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.538155    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:30:30.542662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.542631    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"]
Oct 30 03:30:30.551749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.551722    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:30:30.552344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.552065    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:30:30.552344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.552152    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:30:30.556795 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.556769    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"]
Oct 30 03:30:30.624028 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.623983    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3553944-wrr7l\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.624222 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.624042    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97ts6\" (UniqueName: \"kubernetes.io/projected/afb0ce9d-967a-4ca3-bd70-e502e5651a68-kube-api-access-97ts6\") pod \"maas-api-revoke-sub-a1ffeef5-lc4ds\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:30.624286 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.624221    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxtj\" (UniqueName: \"kubernetes.io/projected/88e33731-0cc6-4e77-89fc-7ee4cb97746a-kube-api-access-twxtj\") pod \"maas-api-revoke-sub-b3553944-wrr7l\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.624335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.624280    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ffeef5-lc4ds\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:30.624467 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.624419    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3553944-wrr7l\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.624525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.624486    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ffeef5-lc4ds\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:30.725694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.725652    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxtj\" (UniqueName: \"kubernetes.io/projected/88e33731-0cc6-4e77-89fc-7ee4cb97746a-kube-api-access-twxtj\") pod \"maas-api-revoke-sub-b3553944-wrr7l\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.725998 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.725976    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ffeef5-lc4ds\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:30.726165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.726140    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3553944-wrr7l\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.726316 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.726176    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ffeef5-lc4ds\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:30.726316 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.726294    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3553944-wrr7l\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.726475 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.726321    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97ts6\" (UniqueName: \"kubernetes.io/projected/afb0ce9d-967a-4ca3-bd70-e502e5651a68-kube-api-access-97ts6\") pod \"maas-api-revoke-sub-a1ffeef5-lc4ds\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:30.727266 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.727232    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ffeef5-lc4ds\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:30.727768 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.727743    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3553944-wrr7l\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.729537 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.729493    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ffeef5-lc4ds\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:30.742505 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.742372    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3553944-wrr7l\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.746337 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.746305    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxtj\" (UniqueName: \"kubernetes.io/projected/88e33731-0cc6-4e77-89fc-7ee4cb97746a-kube-api-access-twxtj\") pod \"maas-api-revoke-sub-b3553944-wrr7l\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.751413 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.751353    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97ts6\" (UniqueName: \"kubernetes.io/projected/afb0ce9d-967a-4ca3-bd70-e502e5651a68-kube-api-access-97ts6\") pod \"maas-api-revoke-sub-a1ffeef5-lc4ds\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:30.836936 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.836880    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:30.848515 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:30.848483    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:31.146374 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:31.146341    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"]
Oct 30 03:30:31.148756 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:30:31.148721    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e33731_0cc6_4e77_89fc_7ee4cb97746a.slice/crio-c4d8bf35c92ef7e0b4cc4d47d2fd4071148f160028a0770067f748eef0413126 WatchSource:0}: Error finding container c4d8bf35c92ef7e0b4cc4d47d2fd4071148f160028a0770067f748eef0413126: Status 404 returned error can't find the container with id c4d8bf35c92ef7e0b4cc4d47d2fd4071148f160028a0770067f748eef0413126
Oct 30 03:30:31.160684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:31.160656    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"]
Oct 30 03:30:31.163771 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:30:31.163750    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb0ce9d_967a_4ca3_bd70_e502e5651a68.slice/crio-bbcca068336367e7a78b4dce6b9d76d2d9f30dd2e8f0374d881e389517d29a34 WatchSource:0}: Error finding container bbcca068336367e7a78b4dce6b9d76d2d9f30dd2e8f0374d881e389517d29a34: Status 404 returned error can't find the container with id bbcca068336367e7a78b4dce6b9d76d2d9f30dd2e8f0374d881e389517d29a34
Oct 30 03:30:31.699447 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:31.699387    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds" event={"ID":"afb0ce9d-967a-4ca3-bd70-e502e5651a68","Type":"ContainerStarted","Data":"2666579ba37c076ef1461972884728a289b26fd676897c53c7b2dabeff1a0cc5"}
Oct 30 03:30:31.699902 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:31.699512    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds" event={"ID":"afb0ce9d-967a-4ca3-bd70-e502e5651a68","Type":"ContainerStarted","Data":"bbcca068336367e7a78b4dce6b9d76d2d9f30dd2e8f0374d881e389517d29a34"}
Oct 30 03:30:31.701309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:31.701272    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l" event={"ID":"88e33731-0cc6-4e77-89fc-7ee4cb97746a","Type":"ContainerStarted","Data":"dd97485182d1359a4161f2d62d84921fe2eb3f18d5bcbee11b9dc5c2ca97fa83"}
Oct 30 03:30:31.701489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:31.701330    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l" event={"ID":"88e33731-0cc6-4e77-89fc-7ee4cb97746a","Type":"ContainerStarted","Data":"c4d8bf35c92ef7e0b4cc4d47d2fd4071148f160028a0770067f748eef0413126"}
Oct 30 03:30:31.736388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:31.736265    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds" podStartSLOduration=1.736249948 podStartE2EDuration="1.736249948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:31.734998131 +0000 UTC m=+2969.659206318" watchObservedRunningTime="2026-10-30 03:30:31.736249948 +0000 UTC m=+2969.660458179"
Oct 30 03:30:31.774142 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:31.774085    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l" podStartSLOduration=1.774069527 podStartE2EDuration="1.774069527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:31.773623694 +0000 UTC m=+2969.697831883" watchObservedRunningTime="2026-10-30 03:30:31.774069527 +0000 UTC m=+2969.698277714"
Oct 30 03:30:32.709861 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:32.709823    4278 generic.go:356] "Generic (PLEG): container finished" podID="88e33731-0cc6-4e77-89fc-7ee4cb97746a" containerID="dd97485182d1359a4161f2d62d84921fe2eb3f18d5bcbee11b9dc5c2ca97fa83" exitCode=0
Oct 30 03:30:32.710388 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:32.709917    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l" event={"ID":"88e33731-0cc6-4e77-89fc-7ee4cb97746a","Type":"ContainerDied","Data":"dd97485182d1359a4161f2d62d84921fe2eb3f18d5bcbee11b9dc5c2ca97fa83"}
Oct 30 03:30:32.711700 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:32.711668    4278 generic.go:356] "Generic (PLEG): container finished" podID="afb0ce9d-967a-4ca3-bd70-e502e5651a68" containerID="2666579ba37c076ef1461972884728a289b26fd676897c53c7b2dabeff1a0cc5" exitCode=0
Oct 30 03:30:32.711817 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:32.711740    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds" event={"ID":"afb0ce9d-967a-4ca3-bd70-e502e5651a68","Type":"ContainerDied","Data":"2666579ba37c076ef1461972884728a289b26fd676897c53c7b2dabeff1a0cc5"}
Oct 30 03:30:33.865251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.865218    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:33.897789 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.897759    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:33.963301 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.963267    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb0ce9d-967a-4ca3-bd70-e502e5651a68-kube-api-access-97ts6\" (UniqueName: \"kubernetes.io/projected/afb0ce9d-967a-4ca3-bd70-e502e5651a68-kube-api-access-97ts6\") pod \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") "
Oct 30 03:30:33.963547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.963327    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-openshift-ca\") pod \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") "
Oct 30 03:30:33.963547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.963358    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e33731-0cc6-4e77-89fc-7ee4cb97746a-kube-api-access-twxtj\" (UniqueName: \"kubernetes.io/projected/88e33731-0cc6-4e77-89fc-7ee4cb97746a-kube-api-access-twxtj\") pod \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") "
Oct 30 03:30:33.963547 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.963399    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-xks-ca\") pod \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") "
Oct 30 03:30:33.963721 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.963600    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-openshift-ca\") pod \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\" (UID: \"afb0ce9d-967a-4ca3-bd70-e502e5651a68\") "
Oct 30 03:30:33.963721 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.963644    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-xks-ca\") pod \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\" (UID: \"88e33731-0cc6-4e77-89fc-7ee4cb97746a\") "
Oct 30 03:30:33.963839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.963764    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-openshift-ca" pod "88e33731-0cc6-4e77-89fc-7ee4cb97746a" (UID: "88e33731-0cc6-4e77-89fc-7ee4cb97746a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:33.964141 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.964092    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-openshift-ca" pod "afb0ce9d-967a-4ca3-bd70-e502e5651a68" (UID: "afb0ce9d-967a-4ca3-bd70-e502e5651a68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:33.964280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.964161    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:33.965836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.965803    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-xks-ca" pod "88e33731-0cc6-4e77-89fc-7ee4cb97746a" (UID: "88e33731-0cc6-4e77-89fc-7ee4cb97746a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:33.966492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.966460    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-xks-ca" pod "afb0ce9d-967a-4ca3-bd70-e502e5651a68" (UID: "afb0ce9d-967a-4ca3-bd70-e502e5651a68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:33.966596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.966505    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e33731-0cc6-4e77-89fc-7ee4cb97746a-kube-api-access-twxtj" pod "88e33731-0cc6-4e77-89fc-7ee4cb97746a" (UID: "88e33731-0cc6-4e77-89fc-7ee4cb97746a"). InnerVolumeSpecName "kube-api-access-twxtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:33.966596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:33.966527    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb0ce9d-967a-4ca3-bd70-e502e5651a68-kube-api-access-97ts6" pod "afb0ce9d-967a-4ca3-bd70-e502e5651a68" (UID: "afb0ce9d-967a-4ca3-bd70-e502e5651a68"). InnerVolumeSpecName "kube-api-access-97ts6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:34.064862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.064827    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twxtj\" (UniqueName: \"kubernetes.io/projected/88e33731-0cc6-4e77-89fc-7ee4cb97746a-kube-api-access-twxtj\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:34.064862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.064860    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:34.065114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.064875    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb0ce9d-967a-4ca3-bd70-e502e5651a68-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:34.065114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.064890    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e33731-0cc6-4e77-89fc-7ee4cb97746a-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:34.065114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.064903    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97ts6\" (UniqueName: \"kubernetes.io/projected/afb0ce9d-967a-4ca3-bd70-e502e5651a68-kube-api-access-97ts6\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:34.727115 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.727083    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"
Oct 30 03:30:34.727355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.727113    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds" event={"ID":"afb0ce9d-967a-4ca3-bd70-e502e5651a68","Type":"ContainerDied","Data":"bbcca068336367e7a78b4dce6b9d76d2d9f30dd2e8f0374d881e389517d29a34"}
Oct 30 03:30:34.727355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.727176    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbcca068336367e7a78b4dce6b9d76d2d9f30dd2e8f0374d881e389517d29a34"
Oct 30 03:30:34.728718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.728694    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l" event={"ID":"88e33731-0cc6-4e77-89fc-7ee4cb97746a","Type":"ContainerDied","Data":"c4d8bf35c92ef7e0b4cc4d47d2fd4071148f160028a0770067f748eef0413126"}
Oct 30 03:30:34.728836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.728734    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4d8bf35c92ef7e0b4cc4d47d2fd4071148f160028a0770067f748eef0413126"
Oct 30 03:30:34.728836 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:34.728742    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"
Oct 30 03:30:35.980345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:35.980306    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main" containerID="cri-o://5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d" gracePeriod=45
Oct 30 03:30:36.239942 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.239868    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 03:30:36.307647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.290097    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-model-cache\") pod \"34c5ad60-3ec3-4367-a4ba-304411c1595f\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") "
Oct 30 03:30:36.307647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.290170    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-tmp-dir\") pod \"34c5ad60-3ec3-4367-a4ba-304411c1595f\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") "
Oct 30 03:30:36.307647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.290254    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-dshm\") pod \"34c5ad60-3ec3-4367-a4ba-304411c1595f\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") "
Oct 30 03:30:36.307647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.290338    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-home\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-home\") pod \"34c5ad60-3ec3-4367-a4ba-304411c1595f\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") "
Oct 30 03:30:36.307647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.290437    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c5ad60-3ec3-4367-a4ba-304411c1595f-kube-api-access-jvqdn\" (UniqueName: \"kubernetes.io/projected/34c5ad60-3ec3-4367-a4ba-304411c1595f-kube-api-access-jvqdn\") pod \"34c5ad60-3ec3-4367-a4ba-304411c1595f\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") "
Oct 30 03:30:36.307647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.290504    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34c5ad60-3ec3-4367-a4ba-304411c1595f-tls-certs\" (UniqueName: \"kubernetes.io/secret/34c5ad60-3ec3-4367-a4ba-304411c1595f-tls-certs\") pod \"34c5ad60-3ec3-4367-a4ba-304411c1595f\" (UID: \"34c5ad60-3ec3-4367-a4ba-304411c1595f\") "
Oct 30 03:30:36.313502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.296620    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-home" pod "34c5ad60-3ec3-4367-a4ba-304411c1595f" (UID: "34c5ad60-3ec3-4367-a4ba-304411c1595f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.313502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.306476    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-model-cache" pod "34c5ad60-3ec3-4367-a4ba-304411c1595f" (UID: "34c5ad60-3ec3-4367-a4ba-304411c1595f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.313502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.306916    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-tmp-dir" pod "34c5ad60-3ec3-4367-a4ba-304411c1595f" (UID: "34c5ad60-3ec3-4367-a4ba-304411c1595f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.313502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.311740    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-dshm" pod "34c5ad60-3ec3-4367-a4ba-304411c1595f" (UID: "34c5ad60-3ec3-4367-a4ba-304411c1595f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.321456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.317772    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.321456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.317809    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.321456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.317852    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.321456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.317868    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34c5ad60-3ec3-4367-a4ba-304411c1595f-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.335051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.332111    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c5ad60-3ec3-4367-a4ba-304411c1595f-kube-api-access-jvqdn" pod "34c5ad60-3ec3-4367-a4ba-304411c1595f" (UID: "34c5ad60-3ec3-4367-a4ba-304411c1595f"). InnerVolumeSpecName "kube-api-access-jvqdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:36.335051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.332213    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34c5ad60-3ec3-4367-a4ba-304411c1595f-tls-certs" pod "34c5ad60-3ec3-4367-a4ba-304411c1595f" (UID: "34c5ad60-3ec3-4367-a4ba-304411c1595f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:36.418470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.418407    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvqdn\" (UniqueName: \"kubernetes.io/projected/34c5ad60-3ec3-4367-a4ba-304411c1595f-kube-api-access-jvqdn\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.418470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.418473    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34c5ad60-3ec3-4367-a4ba-304411c1595f-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.744791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.744747    4278 generic.go:356] "Generic (PLEG): container finished" podID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerID="5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d" exitCode=0
Oct 30 03:30:36.745039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.744827    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"
Oct 30 03:30:36.745039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.744830    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" event={"ID":"34c5ad60-3ec3-4367-a4ba-304411c1595f","Type":"ContainerDied","Data":"5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d"}
Oct 30 03:30:36.745039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.744885    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9" event={"ID":"34c5ad60-3ec3-4367-a4ba-304411c1595f","Type":"ContainerDied","Data":"853e4fb4a2f548d05184b1870be022c9432fb1a1bfcc80407f38ff9997efa101"}
Oct 30 03:30:36.745039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.744919    4278 scope.go:122] "RemoveContainer" containerID="5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d"
Oct 30 03:30:36.759867 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.759840    4278 scope.go:122] "RemoveContainer" containerID="5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d"
Oct 30 03:30:36.760329 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:36.760302    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d\": container with ID starting with 5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d not found: ID does not exist" containerID="5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d"
Oct 30 03:30:36.760453 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.760342    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d"} err="failed to get container status \"5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d\": rpc error: code = NotFound desc = could not find container \"5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d\": container with ID starting with 5cdb9a30bff18c7b791e79c212488ca4b57140b374ca5d66af0503edef7ada2d not found: ID does not exist"
Oct 30 03:30:36.796131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.796095    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"]
Oct 30 03:30:36.800152 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:36.800120    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/simulator-w5-f7d1a1-kserve-865766759d-f9nd9"]
Oct 30 03:30:38.198527 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:38.198491    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" path="/var/lib/kubelet/pods/34c5ad60-3ec3-4367-a4ba-304411c1595f/volumes"
Oct 30 03:30:41.514385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:41.514344    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:30:51.592475 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:51.592417    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:30:53.661998 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:53.661958    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"]
Oct 30 03:30:53.673267 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:53.673233    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7d1a1-htwm7"]
Oct 30 03:30:54.190681 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:30:54.190644    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:30:54.198880 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:30:54.198848    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be0cbd8d-88d9-4edf-9131-ce6627a8f2fa" path="/var/lib/kubelet/pods/be0cbd8d-88d9-4edf-9131-ce6627a8f2fa/volumes"
Oct 30 03:31:00.785026 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:00.784964    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"]
Oct 30 03:31:00.785661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:00.785627    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy" containerID="cri-o://c9adb7167f53fac1f0708688857383d148e727a1b671c6360ffedfd92e34e7a9" gracePeriod=30
Oct 30 03:31:04.619150 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:04.619103    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:31:05.293205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:05.293167    4278 scope.go:122] "RemoveContainer" containerID="79ad6bf0348b25c6dbf54033cab6132b6619ed8de1929c94e965b5ccb5fff7f2"
Oct 30 03:31:05.303941 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:05.303913    4278 scope.go:122] "RemoveContainer" containerID="1c618083b57acd80e74b7ac2e7ce8b5ae9dc508ac6137265063e03fde7d340f0"
Oct 30 03:31:05.313662 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:05.313635    4278 scope.go:122] "RemoveContainer" containerID="b297ec6c2fb4c5df7a1343d0babff859b40d3f23cf8cdfc58fa070f4641b7ae1"
Oct 30 03:31:05.325603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:05.325580    4278 scope.go:122] "RemoveContainer" containerID="7126ee82c494de2da84eef14b4a3b7b447162aab54e2dcf0b49359a256573631"
Oct 30 03:31:05.335595 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:05.335573    4278 scope.go:122] "RemoveContainer" containerID="11961fbdb924d1c3fa8217f705b063fa20548d219d3924c2d653f8f5af3160aa"
Oct 30 03:31:05.958638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:05.958601    4278 generic.go:356] "Generic (PLEG): container finished" podID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerID="c9adb7167f53fac1f0708688857383d148e727a1b671c6360ffedfd92e34e7a9" exitCode=0
Oct 30 03:31:05.959054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:05.958674    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" event={"ID":"6abb2acf-b64c-48d4-908a-2394f98f974f","Type":"ContainerDied","Data":"c9adb7167f53fac1f0708688857383d148e727a1b671c6360ffedfd92e34e7a9"}
Oct 30 03:31:06.448977 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.448951    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 03:31:06.520939 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.520905    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abb2acf-b64c-48d4-908a-2394f98f974f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abb2acf-b64c-48d4-908a-2394f98f974f-istiod-ca-cert\") pod \"6abb2acf-b64c-48d4-908a-2394f98f974f\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") "
Oct 30 03:31:06.521146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.520959    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-data\") pod \"6abb2acf-b64c-48d4-908a-2394f98f974f\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") "
Oct 30 03:31:06.521146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.520983    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-socket\") pod \"6abb2acf-b64c-48d4-908a-2394f98f974f\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") "
Oct 30 03:31:06.521146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521001    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-certs\") pod \"6abb2acf-b64c-48d4-908a-2394f98f974f\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") "
Oct 30 03:31:06.521146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521064    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-kube-api-access-mznm2\" (UniqueName: \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-kube-api-access-mznm2\") pod \"6abb2acf-b64c-48d4-908a-2394f98f974f\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") "
Oct 30 03:31:06.521146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521097    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-envoy\") pod \"6abb2acf-b64c-48d4-908a-2394f98f974f\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") "
Oct 30 03:31:06.521146 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521131    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-credential-socket\") pod \"6abb2acf-b64c-48d4-908a-2394f98f974f\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") "
Oct 30 03:31:06.521532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521183    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-token\" (UniqueName: \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-token\") pod \"6abb2acf-b64c-48d4-908a-2394f98f974f\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") "
Oct 30 03:31:06.521532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521229    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-podinfo\") pod \"6abb2acf-b64c-48d4-908a-2394f98f974f\" (UID: \"6abb2acf-b64c-48d4-908a-2394f98f974f\") "
Oct 30 03:31:06.521532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521365    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-socket" pod "6abb2acf-b64c-48d4-908a-2394f98f974f" (UID: "6abb2acf-b64c-48d4-908a-2394f98f974f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:06.521532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521371    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-certs" pod "6abb2acf-b64c-48d4-908a-2394f98f974f" (UID: "6abb2acf-b64c-48d4-908a-2394f98f974f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:06.521532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521386    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-data" pod "6abb2acf-b64c-48d4-908a-2394f98f974f" (UID: "6abb2acf-b64c-48d4-908a-2394f98f974f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:06.521532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521500    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-credential-socket" pod "6abb2acf-b64c-48d4-908a-2394f98f974f" (UID: "6abb2acf-b64c-48d4-908a-2394f98f974f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:06.521532 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521518    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abb2acf-b64c-48d4-908a-2394f98f974f-istiod-ca-cert" pod "6abb2acf-b64c-48d4-908a-2394f98f974f" (UID: "6abb2acf-b64c-48d4-908a-2394f98f974f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:06.521889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521708    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.521889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521731    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abb2acf-b64c-48d4-908a-2394f98f974f-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.521889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521746    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.521889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521760    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.521889 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.521774    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.523698 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.523673    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-envoy" pod "6abb2acf-b64c-48d4-908a-2394f98f974f" (UID: "6abb2acf-b64c-48d4-908a-2394f98f974f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:06.523919 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.523895    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-token" pod "6abb2acf-b64c-48d4-908a-2394f98f974f" (UID: "6abb2acf-b64c-48d4-908a-2394f98f974f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:06.523989 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.523952    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-kube-api-access-mznm2" pod "6abb2acf-b64c-48d4-908a-2394f98f974f" (UID: "6abb2acf-b64c-48d4-908a-2394f98f974f"). InnerVolumeSpecName "kube-api-access-mznm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:06.524031 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.524006    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-podinfo" pod "6abb2acf-b64c-48d4-908a-2394f98f974f" (UID: "6abb2acf-b64c-48d4-908a-2394f98f974f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:06.622894 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.622800    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mznm2\" (UniqueName: \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-kube-api-access-mznm2\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.622894 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.622834    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.622894 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.622845    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.622894 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.622855    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abb2acf-b64c-48d4-908a-2394f98f974f-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.966732 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.966649    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"
Oct 30 03:31:06.966732 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.966676    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd" event={"ID":"6abb2acf-b64c-48d4-908a-2394f98f974f","Type":"ContainerDied","Data":"fe6bd1ee7c69f1a815ba354f705315661d60bf3b538a57ae6edb0bae09c2a16f"}
Oct 30 03:31:06.967255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:06.966769    4278 scope.go:122] "RemoveContainer" containerID="c9adb7167f53fac1f0708688857383d148e727a1b671c6360ffedfd92e34e7a9"
Oct 30 03:31:07.029266 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:07.029223    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"]
Oct 30 03:31:07.031932 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:07.031899    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rbgnd"]
Oct 30 03:31:08.198281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:08.198243    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" path="/var/lib/kubelet/pods/6abb2acf-b64c-48d4-908a-2394f98f974f/volumes"
Oct 30 03:31:20.852950 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.852839    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"]
Oct 30 03:31:20.853465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853395    4278 cpu_manager.go:414] "containerMap: removing container" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy"
Oct 30 03:31:20.853465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853413    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy"
Oct 30 03:31:20.853465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853440    4278 cpu_manager.go:414] "containerMap: removing container" podUID="88e33731-0cc6-4e77-89fc-7ee4cb97746a" containerName="revoke-keys"
Oct 30 03:31:20.853465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853446    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e33731-0cc6-4e77-89fc-7ee4cb97746a" containerName="revoke-keys"
Oct 30 03:31:20.853465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853455    4278 cpu_manager.go:414] "containerMap: removing container" podUID="afb0ce9d-967a-4ca3-bd70-e502e5651a68" containerName="revoke-keys"
Oct 30 03:31:20.853465 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853461    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb0ce9d-967a-4ca3-bd70-e502e5651a68" containerName="revoke-keys"
Oct 30 03:31:20.853718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853479    4278 cpu_manager.go:414] "containerMap: removing container" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main"
Oct 30 03:31:20.853718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853484    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main"
Oct 30 03:31:20.853718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853561    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abb2acf-b64c-48d4-908a-2394f98f974f" containerName="istio-proxy"
Oct 30 03:31:20.853718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853572    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e33731-0cc6-4e77-89fc-7ee4cb97746a" containerName="revoke-keys"
Oct 30 03:31:20.853718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853578    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb0ce9d-967a-4ca3-bd70-e502e5651a68" containerName="revoke-keys"
Oct 30 03:31:20.853718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.853587    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c5ad60-3ec3-4367-a4ba-304411c1595f" containerName="main"
Oct 30 03:31:20.858132 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.858111    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:20.883871 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.883839    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:31:20.884077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.884054    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:31:20.887667 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.887633    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"]
Oct 30 03:31:20.901847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.901817    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:31:20.901847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.901839    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:31:20.971130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.971087    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:20.971315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.971142    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:20.971315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:20.971267    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9qcv\" (UniqueName: \"kubernetes.io/projected/4901ed47-1ff7-4428-8424-6c17b2aac3a5-kube-api-access-k9qcv\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:21.072872 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:21.072833    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9qcv\" (UniqueName: \"kubernetes.io/projected/4901ed47-1ff7-4428-8424-6c17b2aac3a5-kube-api-access-k9qcv\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:21.073107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:21.072942    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:21.073107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:21.072971    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:21.073742 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:21.073721    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:21.075541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:21.075518    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:21.091225 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:21.091193    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9qcv\" (UniqueName: \"kubernetes.io/projected/4901ed47-1ff7-4428-8424-6c17b2aac3a5-kube-api-access-k9qcv\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:21.172114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:21.172022    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:21.470391 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:21.470360    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"]
Oct 30 03:31:21.473363 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:31:21.473329    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4901ed47_1ff7_4428_8424_6c17b2aac3a5.slice/crio-55fa4b1050c51894144e17fe24a1cf3e48ba93ecd37519f0ca86525a5dcbd564 WatchSource:0}: Error finding container 55fa4b1050c51894144e17fe24a1cf3e48ba93ecd37519f0ca86525a5dcbd564: Status 404 returned error can't find the container with id 55fa4b1050c51894144e17fe24a1cf3e48ba93ecd37519f0ca86525a5dcbd564
Oct 30 03:31:22.025796 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:22.025680    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"]
Oct 30 03:31:22.036923 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:22.036888    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56068983-wwr2k"]
Oct 30 03:31:22.073196 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:22.073160    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx" event={"ID":"4901ed47-1ff7-4428-8424-6c17b2aac3a5","Type":"ContainerStarted","Data":"a248b7b4c11dc8dfc685fdd564b6ed87e4a6430e4aef3e9133d610139415bf35"}
Oct 30 03:31:22.073376 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:22.073213    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx" event={"ID":"4901ed47-1ff7-4428-8424-6c17b2aac3a5","Type":"ContainerStarted","Data":"55fa4b1050c51894144e17fe24a1cf3e48ba93ecd37519f0ca86525a5dcbd564"}
Oct 30 03:31:22.113791 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:22.113730    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx" podStartSLOduration=2.113715729 podStartE2EDuration="2.113715729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:22.110225216 +0000 UTC m=+3020.034433404" watchObservedRunningTime="2026-10-30 03:31:22.113715729 +0000 UTC m=+3020.037923916"
Oct 30 03:31:22.197723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:22.197688    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2" path="/var/lib/kubelet/pods/2650cf5e-8ef5-4866-9cfb-af3b1f4bd0d2/volumes"
Oct 30 03:31:23.081535 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:23.081498    4278 generic.go:356] "Generic (PLEG): container finished" podID="4901ed47-1ff7-4428-8424-6c17b2aac3a5" containerID="a248b7b4c11dc8dfc685fdd564b6ed87e4a6430e4aef3e9133d610139415bf35" exitCode=0
Oct 30 03:31:23.081999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:23.081588    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx" event={"ID":"4901ed47-1ff7-4428-8424-6c17b2aac3a5","Type":"ContainerDied","Data":"a248b7b4c11dc8dfc685fdd564b6ed87e4a6430e4aef3e9133d610139415bf35"}
Oct 30 03:31:24.194700 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:31:24.194628    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:31:24.243336 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:24.243303    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:24.407816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:24.407717    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-openshift-ca\") pod \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") "
Oct 30 03:31:24.407816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:24.407778    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4901ed47-1ff7-4428-8424-6c17b2aac3a5-kube-api-access-k9qcv\" (UniqueName: \"kubernetes.io/projected/4901ed47-1ff7-4428-8424-6c17b2aac3a5-kube-api-access-k9qcv\") pod \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") "
Oct 30 03:31:24.407816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:24.407813    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-xks-ca\") pod \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\" (UID: \"4901ed47-1ff7-4428-8424-6c17b2aac3a5\") "
Oct 30 03:31:24.408130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:24.408093    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-openshift-ca" pod "4901ed47-1ff7-4428-8424-6c17b2aac3a5" (UID: "4901ed47-1ff7-4428-8424-6c17b2aac3a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:24.408216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:24.408199    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:24.410077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:24.410047    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-xks-ca" pod "4901ed47-1ff7-4428-8424-6c17b2aac3a5" (UID: "4901ed47-1ff7-4428-8424-6c17b2aac3a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:24.410195 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:24.410118    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4901ed47-1ff7-4428-8424-6c17b2aac3a5-kube-api-access-k9qcv" pod "4901ed47-1ff7-4428-8424-6c17b2aac3a5" (UID: "4901ed47-1ff7-4428-8424-6c17b2aac3a5"). InnerVolumeSpecName "kube-api-access-k9qcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:24.509290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:24.509223    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9qcv\" (UniqueName: \"kubernetes.io/projected/4901ed47-1ff7-4428-8424-6c17b2aac3a5-kube-api-access-k9qcv\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:24.509290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:24.509276    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4901ed47-1ff7-4428-8424-6c17b2aac3a5-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.096510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:25.096468    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx" event={"ID":"4901ed47-1ff7-4428-8424-6c17b2aac3a5","Type":"ContainerDied","Data":"55fa4b1050c51894144e17fe24a1cf3e48ba93ecd37519f0ca86525a5dcbd564"}
Oct 30 03:31:25.096695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:25.096544    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55fa4b1050c51894144e17fe24a1cf3e48ba93ecd37519f0ca86525a5dcbd564"
Oct 30 03:31:25.096695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:25.096484    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"
Oct 30 03:31:32.027819 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:32.027779    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"]
Oct 30 03:31:32.039616 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:32.039563    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25262a2e-tqg2v"]
Oct 30 03:31:32.200524 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:32.200490    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c847b6-9cfe-449f-bf15-184a7b1fa761" path="/var/lib/kubelet/pods/00c847b6-9cfe-449f-bf15-184a7b1fa761/volumes"
Oct 30 03:31:32.861649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:32.861600    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"]
Oct 30 03:31:32.872847 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:32.872811    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f72d5-wmwrx"]
Oct 30 03:31:34.197588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:34.197551    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4901ed47-1ff7-4428-8424-6c17b2aac3a5" path="/var/lib/kubelet/pods/4901ed47-1ff7-4428-8424-6c17b2aac3a5/volumes"
Oct 30 03:31:35.298235 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:35.298199    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:31:40.348748 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.348695    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"]
Oct 30 03:31:40.349584 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.349560    4278 cpu_manager.go:414] "containerMap: removing container" podUID="4901ed47-1ff7-4428-8424-6c17b2aac3a5" containerName="revoke-keys"
Oct 30 03:31:40.349645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.349588    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4901ed47-1ff7-4428-8424-6c17b2aac3a5" containerName="revoke-keys"
Oct 30 03:31:40.349704 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.349694    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="4901ed47-1ff7-4428-8424-6c17b2aac3a5" containerName="revoke-keys"
Oct 30 03:31:40.353341 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.353321    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.379724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.379685    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:31:40.379724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.379685    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:31:40.382243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.382213    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"]
Oct 30 03:31:40.454108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.454070    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:31:40.454316 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.454120    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:31:40.478317 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.478279    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v78k\" (UniqueName: \"kubernetes.io/projected/8daa1444-4196-45d8-8939-d9c1d4efd651-kube-api-access-8v78k\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.478501 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.478335    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.478553 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.478529    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.579657 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.579621    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.579876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.579712    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v78k\" (UniqueName: \"kubernetes.io/projected/8daa1444-4196-45d8-8939-d9c1d4efd651-kube-api-access-8v78k\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.579876 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.579769    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.580456 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.580405    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.582268 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.582232    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.598802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.598731    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v78k\" (UniqueName: \"kubernetes.io/projected/8daa1444-4196-45d8-8939-d9c1d4efd651-kube-api-access-8v78k\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.666653 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.666619    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:40.965550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:40.965466    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"]
Oct 30 03:31:40.969063 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:31:40.969031    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8daa1444_4196_45d8_8939_d9c1d4efd651.slice/crio-edb8af3474ef42d737ecd05a0ef7fe11ef7fad91843d916b8bb7c34c09fa64fa WatchSource:0}: Error finding container edb8af3474ef42d737ecd05a0ef7fe11ef7fad91843d916b8bb7c34c09fa64fa: Status 404 returned error can't find the container with id edb8af3474ef42d737ecd05a0ef7fe11ef7fad91843d916b8bb7c34c09fa64fa
Oct 30 03:31:41.202903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:41.202859    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg" event={"ID":"8daa1444-4196-45d8-8939-d9c1d4efd651","Type":"ContainerStarted","Data":"edb8af3474ef42d737ecd05a0ef7fe11ef7fad91843d916b8bb7c34c09fa64fa"}
Oct 30 03:31:42.210309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:42.210224    4278 generic.go:356] "Generic (PLEG): container finished" podID="8daa1444-4196-45d8-8939-d9c1d4efd651" containerID="765562c6a80adfa19aefa814cc3ca944c4df24d6fda0a6948194d0e6b2b03987" exitCode=0
Oct 30 03:31:42.210309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:42.210294    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg" event={"ID":"8daa1444-4196-45d8-8939-d9c1d4efd651","Type":"ContainerDied","Data":"765562c6a80adfa19aefa814cc3ca944c4df24d6fda0a6948194d0e6b2b03987"}
Oct 30 03:31:43.352086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:43.352060    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:43.404853 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:43.404826    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-xks-ca\") pod \"8daa1444-4196-45d8-8939-d9c1d4efd651\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") "
Oct 30 03:31:43.405024 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:43.404914    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-openshift-ca\") pod \"8daa1444-4196-45d8-8939-d9c1d4efd651\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") "
Oct 30 03:31:43.405094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:43.405054    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8daa1444-4196-45d8-8939-d9c1d4efd651-kube-api-access-8v78k\" (UniqueName: \"kubernetes.io/projected/8daa1444-4196-45d8-8939-d9c1d4efd651-kube-api-access-8v78k\") pod \"8daa1444-4196-45d8-8939-d9c1d4efd651\" (UID: \"8daa1444-4196-45d8-8939-d9c1d4efd651\") "
Oct 30 03:31:43.405312 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:43.405282    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-openshift-ca" pod "8daa1444-4196-45d8-8939-d9c1d4efd651" (UID: "8daa1444-4196-45d8-8939-d9c1d4efd651"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:43.405419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:43.405399    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:43.407079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:43.407046    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-xks-ca" pod "8daa1444-4196-45d8-8939-d9c1d4efd651" (UID: "8daa1444-4196-45d8-8939-d9c1d4efd651"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:43.407170 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:43.407104    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8daa1444-4196-45d8-8939-d9c1d4efd651-kube-api-access-8v78k" pod "8daa1444-4196-45d8-8939-d9c1d4efd651" (UID: "8daa1444-4196-45d8-8939-d9c1d4efd651"). InnerVolumeSpecName "kube-api-access-8v78k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:43.506469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:43.506417    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v78k\" (UniqueName: \"kubernetes.io/projected/8daa1444-4196-45d8-8939-d9c1d4efd651-kube-api-access-8v78k\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:43.506632 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:43.506473    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8daa1444-4196-45d8-8939-d9c1d4efd651-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:31:44.222676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:44.222640    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg" event={"ID":"8daa1444-4196-45d8-8939-d9c1d4efd651","Type":"ContainerDied","Data":"edb8af3474ef42d737ecd05a0ef7fe11ef7fad91843d916b8bb7c34c09fa64fa"}
Oct 30 03:31:44.222830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:44.222687    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edb8af3474ef42d737ecd05a0ef7fe11ef7fad91843d916b8bb7c34c09fa64fa"
Oct 30 03:31:44.222830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:44.222652    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"
Oct 30 03:31:53.968328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:53.968289    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"]
Oct 30 03:31:53.978906 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:53.978879    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-d823d1-ptgqg"]
Oct 30 03:31:54.189295 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:31:54.189262    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:31:54.197688 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:54.197659    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8daa1444-4196-45d8-8939-d9c1d4efd651" path="/var/lib/kubelet/pods/8daa1444-4196-45d8-8939-d9c1d4efd651/volumes"
Oct 30 03:31:56.689512 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:31:56.689478    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:32:01.733051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.733014    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"]
Oct 30 03:32:01.733526 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.733512    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8daa1444-4196-45d8-8939-d9c1d4efd651" containerName="revoke-keys"
Oct 30 03:32:01.733588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.733527    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8daa1444-4196-45d8-8939-d9c1d4efd651" containerName="revoke-keys"
Oct 30 03:32:01.733625 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.733597    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8daa1444-4196-45d8-8939-d9c1d4efd651" containerName="revoke-keys"
Oct 30 03:32:01.738131 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.738115    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:01.756328 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.756296    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"]
Oct 30 03:32:01.764175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.764150    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:32:01.764175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.764160    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:32:01.824805 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.824777    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:32:01.824983 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.824833    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:32:01.890998 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.890962    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrrq\" (UniqueName: \"kubernetes.io/projected/93f3504a-ec9d-4f66-8702-f8d7c883d119-kube-api-access-2wrrq\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:01.890998 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.891005    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:01.891224 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.891075    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:01.992165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.992071    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:01.992165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.992123    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:01.992373 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.992220    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrrq\" (UniqueName: \"kubernetes.io/projected/93f3504a-ec9d-4f66-8702-f8d7c883d119-kube-api-access-2wrrq\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:01.992935 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.992910    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:01.994562 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:01.994532    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:02.010976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:02.010945    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrrq\" (UniqueName: \"kubernetes.io/projected/93f3504a-ec9d-4f66-8702-f8d7c883d119-kube-api-access-2wrrq\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:02.050763 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:02.050729    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:02.347217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:02.347181    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"]
Oct 30 03:32:02.351489 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:32:02.351462    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f3504a_ec9d_4f66_8702_f8d7c883d119.slice/crio-161698e2597556e00734167f23aa39d062871aeac07fbc304bb391d01f6e150b WatchSource:0}: Error finding container 161698e2597556e00734167f23aa39d062871aeac07fbc304bb391d01f6e150b: Status 404 returned error can't find the container with id 161698e2597556e00734167f23aa39d062871aeac07fbc304bb391d01f6e150b
Oct 30 03:32:03.340801 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:03.340764    4278 generic.go:356] "Generic (PLEG): container finished" podID="93f3504a-ec9d-4f66-8702-f8d7c883d119" containerID="23e830dfce2f67ad2decf8b6f1f4cc727fac785f6efbef0c19bbb5df825e1fa4" exitCode=0
Oct 30 03:32:03.341250 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:03.340855    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4" event={"ID":"93f3504a-ec9d-4f66-8702-f8d7c883d119","Type":"ContainerDied","Data":"23e830dfce2f67ad2decf8b6f1f4cc727fac785f6efbef0c19bbb5df825e1fa4"}
Oct 30 03:32:03.341250 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:03.340924    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4" event={"ID":"93f3504a-ec9d-4f66-8702-f8d7c883d119","Type":"ContainerStarted","Data":"161698e2597556e00734167f23aa39d062871aeac07fbc304bb391d01f6e150b"}
Oct 30 03:32:04.487372 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:04.487348    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:04.619586 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:04.619502    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f3504a-ec9d-4f66-8702-f8d7c883d119-kube-api-access-2wrrq\" (UniqueName: \"kubernetes.io/projected/93f3504a-ec9d-4f66-8702-f8d7c883d119-kube-api-access-2wrrq\") pod \"93f3504a-ec9d-4f66-8702-f8d7c883d119\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") "
Oct 30 03:32:04.619743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:04.619586    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-xks-ca\") pod \"93f3504a-ec9d-4f66-8702-f8d7c883d119\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") "
Oct 30 03:32:04.619743 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:04.619631    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-openshift-ca\") pod \"93f3504a-ec9d-4f66-8702-f8d7c883d119\" (UID: \"93f3504a-ec9d-4f66-8702-f8d7c883d119\") "
Oct 30 03:32:04.620139 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:04.620103    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-openshift-ca" pod "93f3504a-ec9d-4f66-8702-f8d7c883d119" (UID: "93f3504a-ec9d-4f66-8702-f8d7c883d119"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:04.621750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:04.621724    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f3504a-ec9d-4f66-8702-f8d7c883d119-kube-api-access-2wrrq" pod "93f3504a-ec9d-4f66-8702-f8d7c883d119" (UID: "93f3504a-ec9d-4f66-8702-f8d7c883d119"). InnerVolumeSpecName "kube-api-access-2wrrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:04.621862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:04.621805    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-xks-ca" pod "93f3504a-ec9d-4f66-8702-f8d7c883d119" (UID: "93f3504a-ec9d-4f66-8702-f8d7c883d119"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:04.721310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:04.721280    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:04.721310 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:04.721307    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wrrq\" (UniqueName: \"kubernetes.io/projected/93f3504a-ec9d-4f66-8702-f8d7c883d119-kube-api-access-2wrrq\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:04.721521 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:04.721317    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f3504a-ec9d-4f66-8702-f8d7c883d119-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:05.353653 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:05.353617    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"
Oct 30 03:32:05.353841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:05.353624    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4" event={"ID":"93f3504a-ec9d-4f66-8702-f8d7c883d119","Type":"ContainerDied","Data":"161698e2597556e00734167f23aa39d062871aeac07fbc304bb391d01f6e150b"}
Oct 30 03:32:05.353841 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:05.353744    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="161698e2597556e00734167f23aa39d062871aeac07fbc304bb391d01f6e150b"
Oct 30 03:32:05.412869 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:05.412839    4278 scope.go:122] "RemoveContainer" containerID="36aa24facc3ce83b47b10e7ad6e93fa3a32b4d10c1e1db5880150fe1ee701eb8"
Oct 30 03:32:05.422093 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:05.422074    4278 scope.go:122] "RemoveContainer" containerID="0517878a61770ee35dfcc1927c63e8da29ab4f94cbd246c52252c68e19a04297"
Oct 30 03:32:05.430387 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:05.430371    4278 scope.go:122] "RemoveContainer" containerID="4e10e3892882a626a8b5fa1ad4e689a096ab546e9afcf11cdbb447c581d973f8"
Oct 30 03:32:05.441178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:05.441157    4278 scope.go:122] "RemoveContainer" containerID="4dca9d636f58f3deb31b0205c85603fba97231689b905b0e845ee8013551c66a"
Oct 30 03:32:13.670298 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:13.670265    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"]
Oct 30 03:32:13.681124 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:13.681096    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-874ab5-xnvc4"]
Oct 30 03:32:14.197685 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:14.197651    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f3504a-ec9d-4f66-8702-f8d7c883d119" path="/var/lib/kubelet/pods/93f3504a-ec9d-4f66-8702-f8d7c883d119/volumes"
Oct 30 03:32:15.482955 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:15.482915    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:32:20.647044 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:20.647008    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"]
Oct 30 03:32:20.647483 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:20.647283    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" podUID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" containerName="istio-proxy" containerID="cri-o://d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c" gracePeriod=30
Oct 30 03:32:22.376300 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.376269    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"]
Oct 30 03:32:22.376759 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.376471    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk" podUID="88f6743d-b3e3-42a2-b1ff-86ae3fd54985" containerName="discovery" containerID="cri-o://7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf" gracePeriod=30
Oct 30 03:32:22.642275 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.642194    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:32:22.677718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.677691    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-ca-configmap\") pod \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") "
Oct 30 03:32:22.677879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.677744    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-token\" (UniqueName: \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-token\") pod \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") "
Oct 30 03:32:22.677879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.677774    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-dns-cert\") pod \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") "
Oct 30 03:32:22.677879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.677844    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-local-certs\") pod \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") "
Oct 30 03:32:22.678058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.677906    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-cacerts\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-cacerts\") pod \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") "
Oct 30 03:32:22.678058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.677966    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-kube-api-access-shlhf\" (UniqueName: \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-kube-api-access-shlhf\") pod \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") "
Oct 30 03:32:22.678058 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.678004    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-kubeconfig\") pod \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\" (UID: \"88f6743d-b3e3-42a2-b1ff-86ae3fd54985\") "
Oct 30 03:32:22.678217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.678092    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-ca-configmap" pod "88f6743d-b3e3-42a2-b1ff-86ae3fd54985" (UID: "88f6743d-b3e3-42a2-b1ff-86ae3fd54985"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:22.678445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.678408    4278 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-ca-configmap\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:22.680237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.680192    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-kube-api-access-shlhf" pod "88f6743d-b3e3-42a2-b1ff-86ae3fd54985" (UID: "88f6743d-b3e3-42a2-b1ff-86ae3fd54985"). InnerVolumeSpecName "kube-api-access-shlhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:22.680335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.680270    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-kubeconfig" pod "88f6743d-b3e3-42a2-b1ff-86ae3fd54985" (UID: "88f6743d-b3e3-42a2-b1ff-86ae3fd54985"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:22.680470 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.680442    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-dns-cert" pod "88f6743d-b3e3-42a2-b1ff-86ae3fd54985" (UID: "88f6743d-b3e3-42a2-b1ff-86ae3fd54985"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:22.680588 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.680507    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-local-certs" pod "88f6743d-b3e3-42a2-b1ff-86ae3fd54985" (UID: "88f6743d-b3e3-42a2-b1ff-86ae3fd54985"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:22.680647 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.680575    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-cacerts" pod "88f6743d-b3e3-42a2-b1ff-86ae3fd54985" (UID: "88f6743d-b3e3-42a2-b1ff-86ae3fd54985"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:22.680878 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.680857    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-token" pod "88f6743d-b3e3-42a2-b1ff-86ae3fd54985" (UID: "88f6743d-b3e3-42a2-b1ff-86ae3fd54985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:22.779390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.779353    4278 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-kubeconfig\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:22.779390 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.779386    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:22.779638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.779400    4278 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-istio-csr-dns-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:22.779638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.779413    4278 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-local-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:22.779638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.779455    4278 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-cacerts\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:22.779638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:22.779467    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shlhf\" (UniqueName: \"kubernetes.io/projected/88f6743d-b3e3-42a2-b1ff-86ae3fd54985-kube-api-access-shlhf\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:23.476572 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:23.476533    4278 generic.go:356] "Generic (PLEG): container finished" podID="88f6743d-b3e3-42a2-b1ff-86ae3fd54985" containerID="7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf" exitCode=0
Oct 30 03:32:23.477018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:23.476589    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"
Oct 30 03:32:23.477018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:23.476619    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk" event={"ID":"88f6743d-b3e3-42a2-b1ff-86ae3fd54985","Type":"ContainerDied","Data":"7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf"}
Oct 30 03:32:23.477018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:23.476674    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk" event={"ID":"88f6743d-b3e3-42a2-b1ff-86ae3fd54985","Type":"ContainerDied","Data":"ac2ca55ba37d22e1c4cd67cda6827cc1d8413c8029ad329def2d10bea240c830"}
Oct 30 03:32:23.477018 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:23.476694    4278 scope.go:122] "RemoveContainer" containerID="7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf"
Oct 30 03:32:23.489249 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:23.489232    4278 scope.go:122] "RemoveContainer" containerID="7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf"
Oct 30 03:32:23.489541 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:32:23.489520    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf\": container with ID starting with 7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf not found: ID does not exist" containerID="7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf"
Oct 30 03:32:23.489594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:23.489550    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf"} err="failed to get container status \"7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf\": rpc error: code = NotFound desc = could not find container \"7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf\": container with ID starting with 7b6bb3d95a9c0388db0efb1703158cf826ad6f7cac9f7db4b8bf111035fbcdbf not found: ID does not exist"
Oct 30 03:32:23.539460 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:23.539420    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"]
Oct 30 03:32:23.550224 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:23.550201    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mhnwk"]
Oct 30 03:32:24.187850 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:32:24.187813    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:32:24.198292 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:24.198265    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88f6743d-b3e3-42a2-b1ff-86ae3fd54985" path="/var/lib/kubelet/pods/88f6743d-b3e3-42a2-b1ff-86ae3fd54985/volumes"
Oct 30 03:32:25.902317 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:25.902290    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:32:26.009713 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.009679    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-credential-socket\") pod \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") "
Oct 30 03:32:26.010025 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.009992    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-kube-api-access-hksgt\" (UniqueName: \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-kube-api-access-hksgt\") pod \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") "
Oct 30 03:32:26.010153 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010047    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-credential-socket" pod "5e25ae83-0654-4360-a8a0-f902f1bd67ad" (UID: "5e25ae83-0654-4360-a8a0-f902f1bd67ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:26.010153 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010101    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-data\") pod \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") "
Oct 30 03:32:26.010243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010160    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-socket\") pod \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") "
Oct 30 03:32:26.010243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010190    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istiod-ca-cert\") pod \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") "
Oct 30 03:32:26.010243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010238    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-podinfo\") pod \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") "
Oct 30 03:32:26.010414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010271    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-certs\") pod \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") "
Oct 30 03:32:26.010414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010308    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-envoy\") pod \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") "
Oct 30 03:32:26.010414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010348    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-data" pod "5e25ae83-0654-4360-a8a0-f902f1bd67ad" (UID: "5e25ae83-0654-4360-a8a0-f902f1bd67ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:26.010414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010349    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-token\" (UniqueName: \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-token\") pod \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\" (UID: \"5e25ae83-0654-4360-a8a0-f902f1bd67ad\") "
Oct 30 03:32:26.010691 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010456    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-socket" pod "5e25ae83-0654-4360-a8a0-f902f1bd67ad" (UID: "5e25ae83-0654-4360-a8a0-f902f1bd67ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:26.010757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010699    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-certs" pod "5e25ae83-0654-4360-a8a0-f902f1bd67ad" (UID: "5e25ae83-0654-4360-a8a0-f902f1bd67ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:26.010757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010721    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istiod-ca-cert" pod "5e25ae83-0654-4360-a8a0-f902f1bd67ad" (UID: "5e25ae83-0654-4360-a8a0-f902f1bd67ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:26.010875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010824    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.010875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010844    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.010875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010858    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.010875 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010872    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.011042 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.010885    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.012202 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.012180    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-kube-api-access-hksgt" pod "5e25ae83-0654-4360-a8a0-f902f1bd67ad" (UID: "5e25ae83-0654-4360-a8a0-f902f1bd67ad"). InnerVolumeSpecName "kube-api-access-hksgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:26.012508 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.012490    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-podinfo" pod "5e25ae83-0654-4360-a8a0-f902f1bd67ad" (UID: "5e25ae83-0654-4360-a8a0-f902f1bd67ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:32:26.012807 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.012785    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-token" pod "5e25ae83-0654-4360-a8a0-f902f1bd67ad" (UID: "5e25ae83-0654-4360-a8a0-f902f1bd67ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:26.012856 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.012799    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-envoy" pod "5e25ae83-0654-4360-a8a0-f902f1bd67ad" (UID: "5e25ae83-0654-4360-a8a0-f902f1bd67ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:26.111608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.111576    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.111608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.111602    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.111608 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.111611    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.111835 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.111620    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hksgt\" (UniqueName: \"kubernetes.io/projected/5e25ae83-0654-4360-a8a0-f902f1bd67ad-kube-api-access-hksgt\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.120172 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.120144    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"]
Oct 30 03:32:26.142908 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.142878    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"]
Oct 30 03:32:26.497022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.496935    4278 generic.go:356] "Generic (PLEG): container finished" podID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" containerID="d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c" exitCode=0
Oct 30 03:32:26.497022 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.497004    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"
Oct 30 03:32:26.497251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.497021    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" event={"ID":"5e25ae83-0654-4360-a8a0-f902f1bd67ad","Type":"ContainerDied","Data":"d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c"}
Oct 30 03:32:26.497251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.497115    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb" event={"ID":"5e25ae83-0654-4360-a8a0-f902f1bd67ad","Type":"ContainerDied","Data":"870bc346ac2465df1654df0aa0274894854f16d20ce24b5e3bdd6c2f155fe26d"}
Oct 30 03:32:26.497251 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.497145    4278 scope.go:122] "RemoveContainer" containerID="d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c"
Oct 30 03:32:26.510352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.510322    4278 scope.go:122] "RemoveContainer" containerID="d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c"
Oct 30 03:32:26.510694 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:32:26.510672    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c\": container with ID starting with d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c not found: ID does not exist" containerID="d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c"
Oct 30 03:32:26.510750 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.510703    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c"} err="failed to get container status \"d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c\": rpc error: code = NotFound desc = could not find container \"d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c\": container with ID starting with d9ef5b1a81b9230d390f48ff89eb7427cefc502bec9c7c2d332c1081c90ba03c not found: ID does not exist"
Oct 30 03:32:26.547698 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.547671    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"]
Oct 30 03:32:26.550863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:26.550839    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-jxjdb"]
Oct 30 03:32:28.198201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:28.198167    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" path="/var/lib/kubelet/pods/5e25ae83-0654-4360-a8a0-f902f1bd67ad/volumes"
Oct 30 03:32:31.744216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:31.744170    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"]
Oct 30 03:32:33.527939 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.527897    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"]
Oct 30 03:32:33.528489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.528321    4278 cpu_manager.go:414] "containerMap: removing container" podUID="88f6743d-b3e3-42a2-b1ff-86ae3fd54985" containerName="discovery"
Oct 30 03:32:33.528489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.528334    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f6743d-b3e3-42a2-b1ff-86ae3fd54985" containerName="discovery"
Oct 30 03:32:33.528489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.528344    4278 cpu_manager.go:414] "containerMap: removing container" podUID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" containerName="istio-proxy"
Oct 30 03:32:33.528489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.528349    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" containerName="istio-proxy"
Oct 30 03:32:33.528489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.528366    4278 cpu_manager.go:414] "containerMap: removing container" podUID="93f3504a-ec9d-4f66-8702-f8d7c883d119" containerName="revoke-keys"
Oct 30 03:32:33.528489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.528371    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f3504a-ec9d-4f66-8702-f8d7c883d119" containerName="revoke-keys"
Oct 30 03:32:33.528489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.528449    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f3504a-ec9d-4f66-8702-f8d7c883d119" containerName="revoke-keys"
Oct 30 03:32:33.528489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.528463    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f6743d-b3e3-42a2-b1ff-86ae3fd54985" containerName="discovery"
Oct 30 03:32:33.528489 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.528470    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e25ae83-0654-4360-a8a0-f902f1bd67ad" containerName="istio-proxy"
Oct 30 03:32:33.533358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.533337    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:33.559417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.559386    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:32:33.559606 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.559523    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:32:33.563211 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.563185    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"]
Oct 30 03:32:33.579782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.579756    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df53da00-jp797\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:33.579914 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.579817    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2knh\" (UniqueName: \"kubernetes.io/projected/8d888547-2902-4190-890d-ea362e6b8b03-kube-api-access-t2knh\") pod \"maas-api-revoke-sub-df53da00-jp797\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:33.579975 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.579940    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df53da00-jp797\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:33.626116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.626083    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:32:33.626116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.626099    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:32:33.681538 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.681498    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df53da00-jp797\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:33.681714 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.681579    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2knh\" (UniqueName: \"kubernetes.io/projected/8d888547-2902-4190-890d-ea362e6b8b03-kube-api-access-t2knh\") pod \"maas-api-revoke-sub-df53da00-jp797\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:33.681714 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.681627    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df53da00-jp797\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:33.682357 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.682335    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df53da00-jp797\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:33.683994 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.683965    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df53da00-jp797\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:33.699816 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.699793    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2knh\" (UniqueName: \"kubernetes.io/projected/8d888547-2902-4190-890d-ea362e6b8b03-kube-api-access-t2knh\") pod \"maas-api-revoke-sub-df53da00-jp797\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:33.847523 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:33.847417    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:34.025280 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:34.025243    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"]
Oct 30 03:32:34.027449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:34.027403    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ec6b54-7gh6q"]
Oct 30 03:32:34.156086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:34.156009    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"]
Oct 30 03:32:34.159398 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:32:34.159359    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d888547_2902_4190_890d_ea362e6b8b03.slice/crio-f2eeb33b5885aabdf56f6ea134d69a7ef27fceecfda95bd5cfd8d02a932812f8 WatchSource:0}: Error finding container f2eeb33b5885aabdf56f6ea134d69a7ef27fceecfda95bd5cfd8d02a932812f8: Status 404 returned error can't find the container with id f2eeb33b5885aabdf56f6ea134d69a7ef27fceecfda95bd5cfd8d02a932812f8
Oct 30 03:32:34.197214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:34.197189    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484aa77b-b879-4fe5-acad-4f1e1d0787d0" path="/var/lib/kubelet/pods/484aa77b-b879-4fe5-acad-4f1e1d0787d0/volumes"
Oct 30 03:32:34.560679 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:34.560635    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797" event={"ID":"8d888547-2902-4190-890d-ea362e6b8b03","Type":"ContainerStarted","Data":"f7aebf493c7df7bed1f440e7d264e04c22b61a60ec882e7bb0db4d57697f85df"}
Oct 30 03:32:34.561127 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:34.560735    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797" event={"ID":"8d888547-2902-4190-890d-ea362e6b8b03","Type":"ContainerStarted","Data":"f2eeb33b5885aabdf56f6ea134d69a7ef27fceecfda95bd5cfd8d02a932812f8"}
Oct 30 03:32:34.597680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:34.597628    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797" podStartSLOduration=1.5976143710000001 podStartE2EDuration="1.597614371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:32:34.597292578 +0000 UTC m=+3092.521500765" watchObservedRunningTime="2026-10-30 03:32:34.597614371 +0000 UTC m=+3092.521822558"
Oct 30 03:32:35.026776 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:35.026738    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"]
Oct 30 03:32:35.030749 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:35.030714    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"]
Oct 30 03:32:35.037802 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:35.037768    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b286c0-xh7j6"]
Oct 30 03:32:35.040059 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:35.040030    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f25a50e-7v4cz"]
Oct 30 03:32:35.568216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:35.568184    4278 generic.go:356] "Generic (PLEG): container finished" podID="8d888547-2902-4190-890d-ea362e6b8b03" containerID="f7aebf493c7df7bed1f440e7d264e04c22b61a60ec882e7bb0db4d57697f85df" exitCode=0
Oct 30 03:32:35.568659 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:35.568236    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797" event={"ID":"8d888547-2902-4190-890d-ea362e6b8b03","Type":"ContainerDied","Data":"f7aebf493c7df7bed1f440e7d264e04c22b61a60ec882e7bb0db4d57697f85df"}
Oct 30 03:32:36.197916 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.197879    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f43f0a6-77de-47a5-9bdb-4b4763f6eb14" path="/var/lib/kubelet/pods/4f43f0a6-77de-47a5-9bdb-4b4763f6eb14/volumes"
Oct 30 03:32:36.198340 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.198326    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1086f8f-0dde-493c-8fd2-c8fe6e2ca516" path="/var/lib/kubelet/pods/f1086f8f-0dde-493c-8fd2-c8fe6e2ca516/volumes"
Oct 30 03:32:36.715891 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.715856    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:36.744913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.744873    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7" podUID="c749bbed-b584-49e5-9bcb-2b81c8e32fee" containerName="payload-processing" containerID="cri-o://f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4" gracePeriod=25
Oct 30 03:32:36.813696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.813658    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-xks-ca\") pod \"8d888547-2902-4190-890d-ea362e6b8b03\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") "
Oct 30 03:32:36.813696 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.813704    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d888547-2902-4190-890d-ea362e6b8b03-kube-api-access-t2knh\" (UniqueName: \"kubernetes.io/projected/8d888547-2902-4190-890d-ea362e6b8b03-kube-api-access-t2knh\") pod \"8d888547-2902-4190-890d-ea362e6b8b03\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") "
Oct 30 03:32:36.813959 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.813748    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-openshift-ca\") pod \"8d888547-2902-4190-890d-ea362e6b8b03\" (UID: \"8d888547-2902-4190-890d-ea362e6b8b03\") "
Oct 30 03:32:36.814171 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.814145    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-openshift-ca" pod "8d888547-2902-4190-890d-ea362e6b8b03" (UID: "8d888547-2902-4190-890d-ea362e6b8b03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:36.815839 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.815807    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-xks-ca" pod "8d888547-2902-4190-890d-ea362e6b8b03" (UID: "8d888547-2902-4190-890d-ea362e6b8b03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:36.815929 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.815859    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d888547-2902-4190-890d-ea362e6b8b03-kube-api-access-t2knh" pod "8d888547-2902-4190-890d-ea362e6b8b03" (UID: "8d888547-2902-4190-890d-ea362e6b8b03"). InnerVolumeSpecName "kube-api-access-t2knh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:36.915561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.915458    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:36.915561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.915493    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2knh\" (UniqueName: \"kubernetes.io/projected/8d888547-2902-4190-890d-ea362e6b8b03-kube-api-access-t2knh\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:36.915561 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:36.915516    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d888547-2902-4190-890d-ea362e6b8b03-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:37.581842 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:37.581814    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"
Oct 30 03:32:37.582006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:37.581812    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797" event={"ID":"8d888547-2902-4190-890d-ea362e6b8b03","Type":"ContainerDied","Data":"f2eeb33b5885aabdf56f6ea134d69a7ef27fceecfda95bd5cfd8d02a932812f8"}
Oct 30 03:32:37.582006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:37.581942    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2eeb33b5885aabdf56f6ea134d69a7ef27fceecfda95bd5cfd8d02a932812f8"
Oct 30 03:32:41.140860 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.140816    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main" containerID="cri-o://672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf" gracePeriod=45
Oct 30 03:32:41.163024 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.162988    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main" containerID="cri-o://db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab" gracePeriod=45
Oct 30 03:32:41.406705 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.406628    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 03:32:41.436263 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.436236    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 03:32:41.563868 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.563829    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd65ad7e-0c67-473f-8199-5af294755d6c-kube-api-access-mzsjp\" (UniqueName: \"kubernetes.io/projected/fd65ad7e-0c67-473f-8199-5af294755d6c-kube-api-access-mzsjp\") pod \"fd65ad7e-0c67-473f-8199-5af294755d6c\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") "
Oct 30 03:32:41.564066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.563888    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbef8fa2-7fb0-4197-b126-3441d7420772-kube-api-access-xzhwr\" (UniqueName: \"kubernetes.io/projected/fbef8fa2-7fb0-4197-b126-3441d7420772-kube-api-access-xzhwr\") pod \"fbef8fa2-7fb0-4197-b126-3441d7420772\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") "
Oct 30 03:32:41.564066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.563926    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-model-cache\") pod \"fbef8fa2-7fb0-4197-b126-3441d7420772\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") "
Oct 30 03:32:41.564066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.563951    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-tmp-dir\") pod \"fd65ad7e-0c67-473f-8199-5af294755d6c\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") "
Oct 30 03:32:41.564256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564072    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-dshm\") pod \"fd65ad7e-0c67-473f-8199-5af294755d6c\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") "
Oct 30 03:32:41.564256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564133    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-model-cache\") pod \"fd65ad7e-0c67-473f-8199-5af294755d6c\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") "
Oct 30 03:32:41.564256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564186    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbef8fa2-7fb0-4197-b126-3441d7420772-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbef8fa2-7fb0-4197-b126-3441d7420772-tls-certs\") pod \"fbef8fa2-7fb0-4197-b126-3441d7420772\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") "
Oct 30 03:32:41.564256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564226    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-tmp-dir\") pod \"fbef8fa2-7fb0-4197-b126-3441d7420772\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") "
Oct 30 03:32:41.564256 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564244    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-model-cache" pod "fbef8fa2-7fb0-4197-b126-3441d7420772" (UID: "fbef8fa2-7fb0-4197-b126-3441d7420772"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.564602 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564277    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-dshm\") pod \"fbef8fa2-7fb0-4197-b126-3441d7420772\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") "
Oct 30 03:32:41.564602 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564311    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-home\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-home\") pod \"fbef8fa2-7fb0-4197-b126-3441d7420772\" (UID: \"fbef8fa2-7fb0-4197-b126-3441d7420772\") "
Oct 30 03:32:41.564602 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564355    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-home\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-home\") pod \"fd65ad7e-0c67-473f-8199-5af294755d6c\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") "
Oct 30 03:32:41.564602 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564413    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-model-cache" pod "fd65ad7e-0c67-473f-8199-5af294755d6c" (UID: "fd65ad7e-0c67-473f-8199-5af294755d6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.564602 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564439    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd65ad7e-0c67-473f-8199-5af294755d6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd65ad7e-0c67-473f-8199-5af294755d6c-tls-certs\") pod \"fd65ad7e-0c67-473f-8199-5af294755d6c\" (UID: \"fd65ad7e-0c67-473f-8199-5af294755d6c\") "
Oct 30 03:32:41.564925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564847    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.564925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.564870    4278 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-model-cache\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.565151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.565125    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-tmp-dir" pod "fd65ad7e-0c67-473f-8199-5af294755d6c" (UID: "fd65ad7e-0c67-473f-8199-5af294755d6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.565535 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.565420    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-home" pod "fbef8fa2-7fb0-4197-b126-3441d7420772" (UID: "fbef8fa2-7fb0-4197-b126-3441d7420772"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.565695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.565524    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-tmp-dir" pod "fbef8fa2-7fb0-4197-b126-3441d7420772" (UID: "fbef8fa2-7fb0-4197-b126-3441d7420772"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.565801 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.565707    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-home" pod "fd65ad7e-0c67-473f-8199-5af294755d6c" (UID: "fd65ad7e-0c67-473f-8199-5af294755d6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.567112 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.567078    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbef8fa2-7fb0-4197-b126-3441d7420772-kube-api-access-xzhwr" pod "fbef8fa2-7fb0-4197-b126-3441d7420772" (UID: "fbef8fa2-7fb0-4197-b126-3441d7420772"). InnerVolumeSpecName "kube-api-access-xzhwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:41.567385 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.567358    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-dshm" pod "fd65ad7e-0c67-473f-8199-5af294755d6c" (UID: "fd65ad7e-0c67-473f-8199-5af294755d6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.567550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.567502    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd65ad7e-0c67-473f-8199-5af294755d6c-tls-certs" pod "fd65ad7e-0c67-473f-8199-5af294755d6c" (UID: "fd65ad7e-0c67-473f-8199-5af294755d6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:41.567550 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.567526    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd65ad7e-0c67-473f-8199-5af294755d6c-kube-api-access-mzsjp" pod "fd65ad7e-0c67-473f-8199-5af294755d6c" (UID: "fd65ad7e-0c67-473f-8199-5af294755d6c"). InnerVolumeSpecName "kube-api-access-mzsjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:41.567958 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.567930    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbef8fa2-7fb0-4197-b126-3441d7420772-tls-certs" pod "fbef8fa2-7fb0-4197-b126-3441d7420772" (UID: "fbef8fa2-7fb0-4197-b126-3441d7420772"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:41.568063 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.567953    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-dshm" pod "fbef8fa2-7fb0-4197-b126-3441d7420772" (UID: "fbef8fa2-7fb0-4197-b126-3441d7420772"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.607198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.607155    4278 generic.go:356] "Generic (PLEG): container finished" podID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerID="672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf" exitCode=0
Oct 30 03:32:41.607365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.607273    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"
Oct 30 03:32:41.607365 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.607261    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" event={"ID":"fd65ad7e-0c67-473f-8199-5af294755d6c","Type":"ContainerDied","Data":"672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf"}
Oct 30 03:32:41.607502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.607390    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt" event={"ID":"fd65ad7e-0c67-473f-8199-5af294755d6c","Type":"ContainerDied","Data":"3c2da9e2f24319c34f77c0ca21e91f99d3d6f182882229a0a34df25a84b1e11a"}
Oct 30 03:32:41.607502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.607414    4278 scope.go:122] "RemoveContainer" containerID="672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf"
Oct 30 03:32:41.609155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.609024    4278 generic.go:356] "Generic (PLEG): container finished" podID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerID="db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab" exitCode=0
Oct 30 03:32:41.609155 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.609078    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"
Oct 30 03:32:41.609319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.609083    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" event={"ID":"fbef8fa2-7fb0-4197-b126-3441d7420772","Type":"ContainerDied","Data":"db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab"}
Oct 30 03:32:41.609319 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.609202    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s" event={"ID":"fbef8fa2-7fb0-4197-b126-3441d7420772","Type":"ContainerDied","Data":"dc5187011aef18060f8330f880c2c1bb87f03578d630d99fe76763943e913765"}
Oct 30 03:32:41.617604 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.617561    4278 scope.go:122] "RemoveContainer" containerID="672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf"
Oct 30 03:32:41.617986 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:32:41.617964    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf\": container with ID starting with 672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf not found: ID does not exist" containerID="672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf"
Oct 30 03:32:41.618105 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.617999    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf"} err="failed to get container status \"672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf\": rpc error: code = NotFound desc = could not find container \"672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf\": container with ID starting with 672200d0ac6645d6c2a49c48060e35ad4cb348a83bb233a590b5e23513d91dcf not found: ID does not exist"
Oct 30 03:32:41.618105 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.618023    4278 scope.go:122] "RemoveContainer" containerID="db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab"
Oct 30 03:32:41.630284 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.630267    4278 scope.go:122] "RemoveContainer" containerID="db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab"
Oct 30 03:32:41.630850 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:32:41.630828    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab\": container with ID starting with db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab not found: ID does not exist" containerID="db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab"
Oct 30 03:32:41.630954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.630857    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab"} err="failed to get container status \"db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab\": rpc error: code = NotFound desc = could not find container \"db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab\": container with ID starting with db0921ab6a857f0da86ba99600c1b4a69e901c0fb3312adfebcb7d3ed1ddf3ab not found: ID does not exist"
Oct 30 03:32:41.655862 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.655826    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"]
Oct 30 03:32:41.661222 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.661148    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/simulator-w0-aeff1d-kserve-6fc9bc446-nwx2s"]
Oct 30 03:32:41.666015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.665991    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.666015 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.666017    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbef8fa2-7fb0-4197-b126-3441d7420772-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.666186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.666026    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.666186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.666034    4278 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-dshm\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.666186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.666041    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbef8fa2-7fb0-4197-b126-3441d7420772-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.666186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.666048    4278 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-home\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.666186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.666059    4278 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd65ad7e-0c67-473f-8199-5af294755d6c-tls-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.666186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.666072    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzsjp\" (UniqueName: \"kubernetes.io/projected/fd65ad7e-0c67-473f-8199-5af294755d6c-kube-api-access-mzsjp\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.666186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.666082    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzhwr\" (UniqueName: \"kubernetes.io/projected/fbef8fa2-7fb0-4197-b126-3441d7420772-kube-api-access-xzhwr\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.666186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.666091    4278 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd65ad7e-0c67-473f-8199-5af294755d6c-tmp-dir\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.692533 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.692493    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"]
Oct 30 03:32:41.696343 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:41.696308    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-2-w0-aeff1d-kserve-899d4bc49-qshkt"]
Oct 30 03:32:42.198737 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:42.198703    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" path="/var/lib/kubelet/pods/fbef8fa2-7fb0-4197-b126-3441d7420772/volumes"
Oct 30 03:32:42.199322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:42.199301    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" path="/var/lib/kubelet/pods/fd65ad7e-0c67-473f-8199-5af294755d6c/volumes"
Oct 30 03:32:42.518613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:42.518569    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7" podUID="c749bbed-b584-49e5-9bcb-2b81c8e32fee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:45.032145 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:45.032050    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"]
Oct 30 03:32:45.043107 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:45.043070    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a123204f-sjjkd"]
Oct 30 03:32:46.117661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:46.117625    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:32:46.198349 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:46.197912    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42229be7-6a18-45e5-82a2-7a2f3f173836" path="/var/lib/kubelet/pods/42229be7-6a18-45e5-82a2-7a2f3f173836/volumes"
Oct 30 03:32:52.400694 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:52.400654    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:32:52.519404 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:52.519366    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7" podUID="c749bbed-b584-49e5-9bcb-2b81c8e32fee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:54.187306 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:32:54.187270    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:32:56.065848 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:56.065811    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"]
Oct 30 03:32:56.066290 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:56.066108    4278 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy" containerID="cri-o://71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d" gracePeriod=30
Oct 30 03:32:56.991536 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:56.991504    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 03:32:57.010265 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.010237    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c749bbed-b584-49e5-9bcb-2b81c8e32fee-kube-api-access-2n2hb\" (UniqueName: \"kubernetes.io/projected/c749bbed-b584-49e5-9bcb-2b81c8e32fee-kube-api-access-2n2hb\") pod \"c749bbed-b584-49e5-9bcb-2b81c8e32fee\" (UID: \"c749bbed-b584-49e5-9bcb-2b81c8e32fee\") "
Oct 30 03:32:57.010400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.010290    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c749bbed-b584-49e5-9bcb-2b81c8e32fee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c749bbed-b584-49e5-9bcb-2b81c8e32fee-plugins-config-volume\") pod \"c749bbed-b584-49e5-9bcb-2b81c8e32fee\" (UID: \"c749bbed-b584-49e5-9bcb-2b81c8e32fee\") "
Oct 30 03:32:57.010718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.010697    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c749bbed-b584-49e5-9bcb-2b81c8e32fee-plugins-config-volume" pod "c749bbed-b584-49e5-9bcb-2b81c8e32fee" (UID: "c749bbed-b584-49e5-9bcb-2b81c8e32fee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:57.012255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.012234    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c749bbed-b584-49e5-9bcb-2b81c8e32fee-kube-api-access-2n2hb" pod "c749bbed-b584-49e5-9bcb-2b81c8e32fee" (UID: "c749bbed-b584-49e5-9bcb-2b81c8e32fee"). InnerVolumeSpecName "kube-api-access-2n2hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:57.111038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.111001    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n2hb\" (UniqueName: \"kubernetes.io/projected/c749bbed-b584-49e5-9bcb-2b81c8e32fee-kube-api-access-2n2hb\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:57.111038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.111031    4278 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c749bbed-b584-49e5-9bcb-2b81c8e32fee-plugins-config-volume\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:32:57.710402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.710363    4278 generic.go:356] "Generic (PLEG): container finished" podID="c749bbed-b584-49e5-9bcb-2b81c8e32fee" containerID="f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4" exitCode=0
Oct 30 03:32:57.710585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.710444    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"
Oct 30 03:32:57.710585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.710453    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7" event={"ID":"c749bbed-b584-49e5-9bcb-2b81c8e32fee","Type":"ContainerDied","Data":"f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4"}
Oct 30 03:32:57.710585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.710514    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7" event={"ID":"c749bbed-b584-49e5-9bcb-2b81c8e32fee","Type":"ContainerDied","Data":"fd96ac65f50273360e46e365293b366e9dec89623b1dd482e7a0815d2af62801"}
Oct 30 03:32:57.710585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.710537    4278 scope.go:122] "RemoveContainer" containerID="f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4"
Oct 30 03:32:57.720233 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.720203    4278 scope.go:122] "RemoveContainer" containerID="f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4"
Oct 30 03:32:57.720517 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:32:57.720497    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4\": container with ID starting with f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4 not found: ID does not exist" containerID="f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4"
Oct 30 03:32:57.720603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.720524    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4"} err="failed to get container status \"f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4\": rpc error: code = NotFound desc = could not find container \"f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4\": container with ID starting with f28c002d99c21d9a932909dd20da5440bb9ba7c93554aab91c66c99a84008cf4 not found: ID does not exist"
Oct 30 03:32:57.755825 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:32:57.755801    4278 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc749bbed_b584_49e5_9bcb_2b81c8e32fee.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc749bbed_b584_49e5_9bcb_2b81c8e32fee.slice/crio-fd96ac65f50273360e46e365293b366e9dec89623b1dd482e7a0815d2af62801\": RecentStats: unable to find data in memory cache]"
Oct 30 03:32:57.760778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.760754    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"]
Oct 30 03:32:57.770628 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:57.770604    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-aeff1d-77d496d48d-khlt7"]
Oct 30 03:32:58.197563 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:32:58.197527    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c749bbed-b584-49e5-9bcb-2b81c8e32fee" path="/var/lib/kubelet/pods/c749bbed-b584-49e5-9bcb-2b81c8e32fee/volumes"
Oct 30 03:33:01.313269 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.313240    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 03:33:01.339676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.339594    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-credential-socket\") pod \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") "
Oct 30 03:33:01.339676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.339644    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istiod-ca-cert\") pod \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") "
Oct 30 03:33:01.339676 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.339664    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-kube-api-access-mf6th\" (UniqueName: \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-kube-api-access-mf6th\") pod \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") "
Oct 30 03:33:01.339954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.339688    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-envoy\") pod \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") "
Oct 30 03:33:01.339954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.339711    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-token\" (UniqueName: \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-token\") pod \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") "
Oct 30 03:33:01.340052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.339944    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-credential-socket" pod "b2df6b23-0d39-4222-b9b1-31568f4df8a9" (UID: "b2df6b23-0d39-4222-b9b1-31568f4df8a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:01.340231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.340202    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istiod-ca-cert" pod "b2df6b23-0d39-4222-b9b1-31568f4df8a9" (UID: "b2df6b23-0d39-4222-b9b1-31568f4df8a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:33:01.341969 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.341948    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-token" pod "b2df6b23-0d39-4222-b9b1-31568f4df8a9" (UID: "b2df6b23-0d39-4222-b9b1-31568f4df8a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:33:01.342054 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.341983    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-envoy" pod "b2df6b23-0d39-4222-b9b1-31568f4df8a9" (UID: "b2df6b23-0d39-4222-b9b1-31568f4df8a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:01.342140 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.342118    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-kube-api-access-mf6th" pod "b2df6b23-0d39-4222-b9b1-31568f4df8a9" (UID: "b2df6b23-0d39-4222-b9b1-31568f4df8a9"). InnerVolumeSpecName "kube-api-access-mf6th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:33:01.440201 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440160    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-socket\") pod \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") "
Oct 30 03:33:01.440407 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440209    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-certs\") pod \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") "
Oct 30 03:33:01.440407 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440264    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-podinfo\") pod \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") "
Oct 30 03:33:01.440407 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440309    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-data\") pod \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\" (UID: \"b2df6b23-0d39-4222-b9b1-31568f4df8a9\") "
Oct 30 03:33:01.440603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440501    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-socket" pod "b2df6b23-0d39-4222-b9b1-31568f4df8a9" (UID: "b2df6b23-0d39-4222-b9b1-31568f4df8a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:01.440603 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440514    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-certs" pod "b2df6b23-0d39-4222-b9b1-31568f4df8a9" (UID: "b2df6b23-0d39-4222-b9b1-31568f4df8a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:01.440684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440611    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-data" pod "b2df6b23-0d39-4222-b9b1-31568f4df8a9" (UID: "b2df6b23-0d39-4222-b9b1-31568f4df8a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:01.440684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440658    4278 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-credential-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.440684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440678    4278 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istiod-ca-cert\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.440779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440689    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf6th\" (UniqueName: \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-kube-api-access-mf6th\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.440779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440699    4278 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-envoy\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.440779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440707    4278 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-token\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.440779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440715    4278 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-socket\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.440779 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.440723    4278 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-workload-certs\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.442402 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.442383    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-podinfo" pod "b2df6b23-0d39-4222-b9b1-31568f4df8a9" (UID: "b2df6b23-0d39-4222-b9b1-31568f4df8a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:33:01.541882 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.541849    4278 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-podinfo\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.541882 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.541876    4278 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2df6b23-0d39-4222-b9b1-31568f4df8a9-istio-data\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.734897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.734800    4278 generic.go:356] "Generic (PLEG): container finished" podID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerID="71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d" exitCode=0
Oct 30 03:33:01.734897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.734852    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" event={"ID":"b2df6b23-0d39-4222-b9b1-31568f4df8a9","Type":"ContainerDied","Data":"71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d"}
Oct 30 03:33:01.734897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.734858    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"
Oct 30 03:33:01.734897 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.734893    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w" event={"ID":"b2df6b23-0d39-4222-b9b1-31568f4df8a9","Type":"ContainerDied","Data":"1397603c8454f7f61f1a556e4f890ea27c3d8df63db8f8f071c91176ee7d0b36"}
Oct 30 03:33:01.735261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.734914    4278 scope.go:122] "RemoveContainer" containerID="71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d"
Oct 30 03:33:01.745013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.744992    4278 scope.go:122] "RemoveContainer" containerID="71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d"
Oct 30 03:33:01.745292 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:33:01.745272    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d\": container with ID starting with 71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d not found: ID does not exist" containerID="71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d"
Oct 30 03:33:01.745345 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.745300    4278 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d"} err="failed to get container status \"71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d\": rpc error: code = NotFound desc = could not find container \"71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d\": container with ID starting with 71e55c261f5abc14088479741a6feb44b86441cdb080a07efb3254720906d91d not found: ID does not exist"
Oct 30 03:33:01.784335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.784306    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"]
Oct 30 03:33:01.795471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:01.795439    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-fkm5w"]
Oct 30 03:33:02.197471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:02.197417    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" path="/var/lib/kubelet/pods/b2df6b23-0d39-4222-b9b1-31568f4df8a9/volumes"
Oct 30 03:33:05.481577 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:05.481541    4278 scope.go:122] "RemoveContainer" containerID="49282fd5c3776795498c3f92ee13e26342bfd59777e32ff7ab3c447e6cf0ab14"
Oct 30 03:33:05.495855 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:05.495831    4278 scope.go:122] "RemoveContainer" containerID="bc03c1cbdc0bd87a2ef56a818a0fb3e6ac0e363366a6ce2178fda747d60a3347"
Oct 30 03:33:05.506309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:05.506287    4278 scope.go:122] "RemoveContainer" containerID="f574d2461c5e9d79522c9ffe9592cfd510677b7ca00dad30bc85ddbed5f6b2e4"
Oct 30 03:33:05.514254 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:05.514238    4278 scope.go:122] "RemoveContainer" containerID="6abb5cfe2166f4e5cab752e2a706076b73942fb4b3fa9f2136eff74d78847f1f"
Oct 30 03:33:05.522367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:05.522347    4278 scope.go:122] "RemoveContainer" containerID="4fd35dcfdff1d1d45e2e6c53d9b647a2badf17a10b98083e7ce18db76087ae9e"
Oct 30 03:33:05.529901 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:05.529876    4278 scope.go:122] "RemoveContainer" containerID="64f46d2d3361f9b1e689a3f0431caf9004df7f342d9255f73aa85be8d3558420"
Oct 30 03:33:20.157094 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157053    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"]
Oct 30 03:33:20.157782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157692    4278 cpu_manager.go:414] "containerMap: removing container" podUID="c749bbed-b584-49e5-9bcb-2b81c8e32fee" containerName="payload-processing"
Oct 30 03:33:20.157782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157717    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c749bbed-b584-49e5-9bcb-2b81c8e32fee" containerName="payload-processing"
Oct 30 03:33:20.157782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157735    4278 cpu_manager.go:414] "containerMap: removing container" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main"
Oct 30 03:33:20.157782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157744    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main"
Oct 30 03:33:20.157782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157758    4278 cpu_manager.go:414] "containerMap: removing container" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy"
Oct 30 03:33:20.157782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157767    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy"
Oct 30 03:33:20.158130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157790    4278 cpu_manager.go:414] "containerMap: removing container" podUID="8d888547-2902-4190-890d-ea362e6b8b03" containerName="revoke-keys"
Oct 30 03:33:20.158130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157799    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d888547-2902-4190-890d-ea362e6b8b03" containerName="revoke-keys"
Oct 30 03:33:20.158130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157812    4278 cpu_manager.go:414] "containerMap: removing container" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main"
Oct 30 03:33:20.158130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157819    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main"
Oct 30 03:33:20.158130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157913    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d888547-2902-4190-890d-ea362e6b8b03" containerName="revoke-keys"
Oct 30 03:33:20.158130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157925    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbef8fa2-7fb0-4197-b126-3441d7420772" containerName="main"
Oct 30 03:33:20.158130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157943    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="c749bbed-b584-49e5-9bcb-2b81c8e32fee" containerName="payload-processing"
Oct 30 03:33:20.158130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157952    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd65ad7e-0c67-473f-8199-5af294755d6c" containerName="main"
Oct 30 03:33:20.158130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.157960    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2df6b23-0d39-4222-b9b1-31568f4df8a9" containerName="istio-proxy"
Oct 30 03:33:20.163607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.163583    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.178070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.178051    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wg9w9\""
Oct 30 03:33:20.199288 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.199261    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"]
Oct 30 03:33:20.206943 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.206907    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.207067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.206948    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.207067 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.206990    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.207214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.207079    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.207214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.207109    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55ht\" (UniqueName: \"kubernetes.io/projected/1554e25e-0893-4e42-a08d-2c8b1a0445ba-kube-api-access-l55ht\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.207306 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.207202    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.207306 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.207260    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.207382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.207318    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.207382 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.207377    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.307947 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.307913    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308123 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.307966    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308123 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308076    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308247 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308129    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308247 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308150    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308279    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308351 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308322    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55ht\" (UniqueName: \"kubernetes.io/projected/1554e25e-0893-4e42-a08d-2c8b1a0445ba-kube-api-access-l55ht\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308368    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308372    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308404    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308503 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308470    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308678 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308521    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308762    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.308859 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.308842    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.310735 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.310710    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.310907 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.310890    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.326279 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.326246    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1554e25e-0893-4e42-a08d-2c8b1a0445ba-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.326384 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.326306    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55ht\" (UniqueName: \"kubernetes.io/projected/1554e25e-0893-4e42-a08d-2c8b1a0445ba-kube-api-access-l55ht\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c22dw\" (UID: \"1554e25e-0893-4e42-a08d-2c8b1a0445ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.477159 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.477071    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:20.789899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.789867    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"]
Oct 30 03:33:20.793448 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:33:20.793410    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1554e25e_0893_4e42_a08d_2c8b1a0445ba.slice/crio-27a4b6c7ced0393976e850dd487626baaca87337ed850d4e30f419701190b698 WatchSource:0}: Error finding container 27a4b6c7ced0393976e850dd487626baaca87337ed850d4e30f419701190b698: Status 404 returned error can't find the container with id 27a4b6c7ced0393976e850dd487626baaca87337ed850d4e30f419701190b698
Oct 30 03:33:20.795706 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.795675    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:33:20.795799 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.795740    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:33:20.795799 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.795767    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:33:20.846695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:20.846660    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" event={"ID":"1554e25e-0893-4e42-a08d-2c8b1a0445ba","Type":"ContainerStarted","Data":"27a4b6c7ced0393976e850dd487626baaca87337ed850d4e30f419701190b698"}
Oct 30 03:33:21.854482 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:21.854445    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" event={"ID":"1554e25e-0893-4e42-a08d-2c8b1a0445ba","Type":"ContainerStarted","Data":"3eae3a61078d12cc25c66e5a9e90c527aa5b7c3898d62f791dd75dc3159f0dec"}
Oct 30 03:33:21.931231 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:21.931176    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" podStartSLOduration=1.931159917 podStartE2EDuration="1.931159917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:21.918073257 +0000 UTC m=+3139.842281465" watchObservedRunningTime="2026-10-30 03:33:21.931159917 +0000 UTC m=+3139.855368105"
Oct 30 03:33:22.479936 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:22.479901    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:22.480373 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:22.480348    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c22dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:33:22.480585 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:22.480558    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" podUID="1554e25e-0893-4e42-a08d-2c8b1a0445ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Oct 30 03:33:23.480973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:23.480937    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c22dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:33:23.481568 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:23.481538    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" podUID="1554e25e-0893-4e42-a08d-2c8b1a0445ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Oct 30 03:33:24.210386 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:33:24.210339    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:33:24.483764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:24.483674    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c22dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:33:24.484383 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:24.484348    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" podUID="1554e25e-0893-4e42-a08d-2c8b1a0445ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Oct 30 03:33:25.478449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:25.477918    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c22dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:33:25.478449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:25.477991    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" podUID="1554e25e-0893-4e42-a08d-2c8b1a0445ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Oct 30 03:33:26.483461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:26.477997    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c22dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:33:26.483461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:26.478074    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" podUID="1554e25e-0893-4e42-a08d-2c8b1a0445ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Oct 30 03:33:27.477539 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:27.477498    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c22dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:33:27.477763 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:27.477550    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" podUID="1554e25e-0893-4e42-a08d-2c8b1a0445ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Oct 30 03:33:29.477978 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:29.477930    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c22dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:33:29.478562 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:29.477986    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" podUID="1554e25e-0893-4e42-a08d-2c8b1a0445ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:33:30.026490 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:30.026443    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"]
Oct 30 03:33:30.036354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:30.036323    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5003bd41-rvtmg"]
Oct 30 03:33:30.201948 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:30.201896    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076479d2-e37d-48e8-8f4b-3a681fd5926a" path="/var/lib/kubelet/pods/076479d2-e37d-48e8-8f4b-3a681fd5926a/volumes"
Oct 30 03:33:30.519610 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:30.519575    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c22dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:33:30.520120 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:30.519650    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" podUID="1554e25e-0893-4e42-a08d-2c8b1a0445ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:33:31.025810 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:31.025774    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"]
Oct 30 03:33:31.036081 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:31.036050    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cc055bd-nnb4f"]
Oct 30 03:33:31.089243 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:31.089212    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:31.089697 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:31.089665    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:31.090626 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:31.090608    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw"
Oct 30 03:33:32.197469 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:32.197420    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66960389-d77e-4b24-bf6e-2033ba1f7d21" path="/var/lib/kubelet/pods/66960389-d77e-4b24-bf6e-2033ba1f7d21/volumes"
Oct 30 03:33:35.207863 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.207833    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"]
Oct 30 03:33:35.212988 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.212934    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.242006 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.241979    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"]
Oct 30 03:33:35.370695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.370657    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.370695 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.370697    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.370913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.370725    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.370913 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.370871    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.370987 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.370930    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.371029 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.371002    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnkd4\" (UniqueName: \"kubernetes.io/projected/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-kube-api-access-xnkd4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.371130 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.371114    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.371190 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.371176    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.371233 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.371220    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.472471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472359    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.472471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472409    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.472471 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472452    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.472751 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472487    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.472751 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472524    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.472751 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472568    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.472751 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472621    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.472751 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472664    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnkd4\" (UniqueName: \"kubernetes.io/projected/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-kube-api-access-xnkd4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.472751 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472687    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.473070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472923    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.473070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.472975    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.473070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.473038    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.473329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.473310    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.473418 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.473394    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.475165 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.475137    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.475438 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.475407    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.490999 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.490969    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.491134 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.491115    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnkd4\" (UniqueName: \"kubernetes.io/projected/7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae-kube-api-access-xnkd4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x5p4f\" (UID: \"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.528636 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.528614    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:35.839181 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.839145    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"]
Oct 30 03:33:35.842505 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:33:35.842477    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6b6b3b_6b19_46b6_8d0a_a3e4b6d156ae.slice/crio-3887e4cfa30c3a2d7430b5ac41528f30dc5b540f8c5c53300beccd628bd417ba WatchSource:0}: Error finding container 3887e4cfa30c3a2d7430b5ac41528f30dc5b540f8c5c53300beccd628bd417ba: Status 404 returned error can't find the container with id 3887e4cfa30c3a2d7430b5ac41528f30dc5b540f8c5c53300beccd628bd417ba
Oct 30 03:33:35.844661 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.844629    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:33:35.844740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.844692    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:33:35.844740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.844737    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:33:35.956818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.956774    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f" event={"ID":"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae","Type":"ContainerStarted","Data":"b72911954a2e93608928c9bdf0d6fcd4bfed19d06fc61d117065ca9159af2253"}
Oct 30 03:33:35.957013 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.956827    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f" event={"ID":"7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae","Type":"ContainerStarted","Data":"3887e4cfa30c3a2d7430b5ac41528f30dc5b540f8c5c53300beccd628bd417ba"}
Oct 30 03:33:35.998459 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:35.998391    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f" podStartSLOduration=0.99837479 podStartE2EDuration="998.37479ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:35.996122466 +0000 UTC m=+3153.920330692" watchObservedRunningTime="2026-10-30 03:33:35.99837479 +0000 UTC m=+3153.922582977"
Oct 30 03:33:36.528970 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:36.528935    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:36.530748 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:36.530718    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:33:36.530883 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:36.530777    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f" podUID="7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 03:33:37.529139 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:37.529098    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:33:37.529651 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:37.529157    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f" podUID="7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 03:33:38.529039 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:38.529008    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:33:38.529230 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:38.529057    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f" podUID="7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 03:33:39.529238 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:39.529200    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:33:39.529686 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:39.529256    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f" podUID="7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 03:33:40.530016 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:40.529980    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:33:40.530502 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:40.530037    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f" podUID="7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 03:33:41.529757 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:41.529713    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:33:41.529946 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:41.529770    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f" podUID="7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 03:33:42.528973 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:42.528935    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:33:42.529367 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:42.528994    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f" podUID="7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 03:33:44.129203 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:44.129171    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:44.129680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:44.129533    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:44.130378 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:33:44.130355    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x5p4f"
Oct 30 03:33:54.188695 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:33:54.188656    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:34:05.488363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:05.488328    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:34:05.593283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:05.593250    4278 scope.go:122] "RemoveContainer" containerID="147cff012941a59e39528b023662581efc880e1e18f6495a7f8c10a7879a30b6"
Oct 30 03:34:05.602119 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:05.602100    4278 scope.go:122] "RemoveContainer" containerID="d25910361a30750827f2655417dd20ac61bf1621a0494709a9fee04a6b897958"
Oct 30 03:34:05.610369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:05.610350    4278 scope.go:122] "RemoveContainer" containerID="2338fdbcf0fa0c2028733c780ecea88b1667548031c5ca23778f2b14ea09c78f"
Oct 30 03:34:05.618101 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:05.618078    4278 scope.go:122] "RemoveContainer" containerID="8578c96c8ddca954cb7ba677a0dbb9696b978f315ffc6f061ff10eb4d1038dcd"
Oct 30 03:34:05.626047 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:05.626031    4278 scope.go:122] "RemoveContainer" containerID="30666648d93923e5f21a2d1f7d780259ce2faeed3397a68562444c3b6f98e0ce"
Oct 30 03:34:05.635446 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:05.635411    4278 scope.go:122] "RemoveContainer" containerID="4d575e34f47529b6dda15c2d0239ebc09ca8519d38544a72f146b6fdd07bbc8d"
Oct 30 03:34:05.643169 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:05.643150    4278 scope.go:122] "RemoveContainer" containerID="19cd623ce9206333b87a9036c56cefc1c95e9ce102406d347c2b4f69a3fceafd"
Oct 30 03:34:05.650607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:05.650590    4278 scope.go:122] "RemoveContainer" containerID="3b9663598b1a839a32b9574fca1f87b1535e9ccac6758d74ca75a9520ca4db37"
Oct 30 03:34:22.789400 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:22.789363    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:34:24.194543 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:34:24.194501    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:34:27.108931 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:27.108888    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:34:29.027045 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:29.027003    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"]
Oct 30 03:34:29.037846 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:29.037816    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"]
Oct 30 03:34:29.039746 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:29.039682    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c640902-qjbb4"]
Oct 30 03:34:29.041081 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:29.041056    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31bcf3c4-tpsdm"]
Oct 30 03:34:30.197793 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:30.197760    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e35e402-7c30-4cad-879e-817b569141ef" path="/var/lib/kubelet/pods/1e35e402-7c30-4cad-879e-817b569141ef/volumes"
Oct 30 03:34:30.198226 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:30.198209    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab139dfe-8de9-4fad-864d-3781187c1238" path="/var/lib/kubelet/pods/ab139dfe-8de9-4fad-864d-3781187c1238/volumes"
Oct 30 03:34:35.462134 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.462094    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"]
Oct 30 03:34:35.465669 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.465650    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.492734 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.492705    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"]
Oct 30 03:34:35.530828 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.530797    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphd5\" (UniqueName: \"kubernetes.io/projected/15027569-cb39-41f1-9ce2-428a5cf9b784-kube-api-access-hphd5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.530976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.530832    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.530976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.530866    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.530976 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.530908    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.531091 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.531002    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15027569-cb39-41f1-9ce2-428a5cf9b784-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.531091 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.531027    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.531160 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.531139    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.531194 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.531159    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.531194 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.531183    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.631801 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.631760    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.631801 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.631806    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.631836    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.631863    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphd5\" (UniqueName: \"kubernetes.io/projected/15027569-cb39-41f1-9ce2-428a5cf9b784-kube-api-access-hphd5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.631889    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.631933    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632092 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.632054    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.632135    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15027569-cb39-41f1-9ce2-428a5cf9b784-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.632165    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632358 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.632259    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.632414    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.632469    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.632507    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.632902 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.632880    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15027569-cb39-41f1-9ce2-428a5cf9b784-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.634452 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.634416    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.634553 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.634505    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.651578 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.651555    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphd5\" (UniqueName: \"kubernetes.io/projected/15027569-cb39-41f1-9ce2-428a5cf9b784-kube-api-access-hphd5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.651673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.651629    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15027569-cb39-41f1-9ce2-428a5cf9b784-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8bc4f\" (UID: \"15027569-cb39-41f1-9ce2-428a5cf9b784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:35.778971 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:35.778934    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:36.097386 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:36.097354    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"]
Oct 30 03:34:36.099086 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:34:36.099060    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15027569_cb39_41f1_9ce2_428a5cf9b784.slice/crio-c46b337681fab669e671abcfc16a321df45795e56eadc0d899a5dec739e596c5 WatchSource:0}: Error finding container c46b337681fab669e671abcfc16a321df45795e56eadc0d899a5dec739e596c5: Status 404 returned error can't find the container with id c46b337681fab669e671abcfc16a321df45795e56eadc0d899a5dec739e596c5
Oct 30 03:34:36.101352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:36.101319    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:34:36.101450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:36.101400    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:34:36.101450 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:36.101444    4278 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:34:36.312276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:36.312237    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" event={"ID":"15027569-cb39-41f1-9ce2-428a5cf9b784","Type":"ContainerStarted","Data":"61c4302484090ea0c164f55fc213fa58544e068ca197dd8d50726f137d333067"}
Oct 30 03:34:36.312486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:36.312298    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" event={"ID":"15027569-cb39-41f1-9ce2-428a5cf9b784","Type":"ContainerStarted","Data":"c46b337681fab669e671abcfc16a321df45795e56eadc0d899a5dec739e596c5"}
Oct 30 03:34:36.357699 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:36.357605    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" podStartSLOduration=1.357591093 podStartE2EDuration="1.357591093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:36.356132771 +0000 UTC m=+3214.280340958" watchObservedRunningTime="2026-10-30 03:34:36.357591093 +0000 UTC m=+3214.281799279"
Oct 30 03:34:36.780445 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:36.780024    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:37.785071 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:37.785039    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Oct 30 03:34:37.785500 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:37.785099    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" podUID="15027569-cb39-41f1-9ce2-428a5cf9b784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Oct 30 03:34:38.779819 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:38.779784    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Oct 30 03:34:38.780102 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:38.780074    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" podUID="15027569-cb39-41f1-9ce2-428a5cf9b784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Oct 30 03:34:39.783451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:39.779792    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Oct 30 03:34:39.783451 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:39.779852    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" podUID="15027569-cb39-41f1-9ce2-428a5cf9b784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Oct 30 03:34:40.779824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:40.779786    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Oct 30 03:34:40.780116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:40.780085    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" podUID="15027569-cb39-41f1-9ce2-428a5cf9b784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Oct 30 03:34:41.779624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:41.779582    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Oct 30 03:34:41.780161 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:41.779642    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" podUID="15027569-cb39-41f1-9ce2-428a5cf9b784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Oct 30 03:34:42.639306 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:42.639266    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:34:42.779458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:42.779395    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Oct 30 03:34:42.779631 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:42.779465    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" podUID="15027569-cb39-41f1-9ce2-428a5cf9b784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Oct 30 03:34:44.780248 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:44.780198    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:34:44.780819 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:44.780266    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" podUID="15027569-cb39-41f1-9ce2-428a5cf9b784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:34:45.822580 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:45.822534    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:34:45.825178 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:45.822604    4278 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" podUID="15027569-cb39-41f1-9ce2-428a5cf9b784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:34:46.023003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:46.022957    4278 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:46.023396 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:46.023371    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:46.062104 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:46.062079    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f"
Oct 30 03:34:54.187049 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:34:54.187012    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:34:58.084407 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:34:58.084369    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:35:05.699055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:05.699016    4278 scope.go:122] "RemoveContainer" containerID="5730a665967b9550c2200cb2d8ad6067f9c6d2b9241d8198eabf35f63e9206ed"
Oct 30 03:35:05.708028 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:05.708005    4278 scope.go:122] "RemoveContainer" containerID="33da365a13fda43484786b4a57c13b8eef92dea49f695239c329448631d5071e"
Oct 30 03:35:05.716615 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:05.716591    4278 scope.go:122] "RemoveContainer" containerID="1059dd021e9c8f41f50ad6f03e7790c5e6abbbfc291d1ed321089b15376af1db"
Oct 30 03:35:05.725050 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:05.725020    4278 scope.go:122] "RemoveContainer" containerID="dabb0a273a9ede8151397177edc2de6354805dd83dc50059f2556a7496f70ebe"
Oct 30 03:35:05.733948 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:05.733923    4278 scope.go:122] "RemoveContainer" containerID="d29a88c589e0745b371eed9b4dd54caee5ac146fbcf128dd63905b6b11ec32fb"
Oct 30 03:35:05.742458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:05.742415    4278 scope.go:122] "RemoveContainer" containerID="4a07dbe76d6d8cb6588ac4c16973390eaa4135b8ee1523f7eee5f801e4a5a73b"
Oct 30 03:35:21.037070 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:21.037026    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"]
Oct 30 03:35:21.050344 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:21.050310    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-203899d4-k9lw5"]
Oct 30 03:35:21.879870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:21.879837    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:35:22.197627 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:22.197546    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b20a143-dfe1-479a-9f5e-04e84278c4f5" path="/var/lib/kubelet/pods/5b20a143-dfe1-479a-9f5e-04e84278c4f5/volumes"
Oct 30 03:35:24.256992 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:35:24.256940    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:35:32.199341 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:32.199308    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:35:34.030283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:34.030233    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"]
Oct 30 03:35:34.045573 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:34.045538    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ffeef5-lc4ds"]
Oct 30 03:35:34.045782 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:34.045761    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"]
Oct 30 03:35:34.047173 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:34.047149    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3553944-wrr7l"]
Oct 30 03:35:34.197896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:34.197862    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e33731-0cc6-4e77-89fc-7ee4cb97746a" path="/var/lib/kubelet/pods/88e33731-0cc6-4e77-89fc-7ee4cb97746a/volumes"
Oct 30 03:35:34.198334 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:34.198317    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb0ce9d-967a-4ca3-bd70-e502e5651a68" path="/var/lib/kubelet/pods/afb0ce9d-967a-4ca3-bd70-e502e5651a68/volumes"
Oct 30 03:35:36.406275 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:36.406239    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:35:53.776607 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:53.776521    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:35:54.200681 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:35:54.200582    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:35:58.422492 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:35:58.422448    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:36:05.782479 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:36:05.782416    4278 scope.go:122] "RemoveContainer" containerID="a7a1e54e84375ce54630bb9ea7275fd821412719d89dd0d5b62bd4572d37cdf4"
Oct 30 03:36:05.792299 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:36:05.792275    4278 scope.go:122] "RemoveContainer" containerID="33ce457a6b11c252feab7a56face5414bb6bcfe2ff2b4c8c8e95551ddd3c1cfc"
Oct 30 03:36:05.804025 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:36:05.803998    4278 scope.go:122] "RemoveContainer" containerID="145f57026ea3ed716c45ee4386ceb216c07fe44454cdf3bf89167ac028036d41"
Oct 30 03:36:05.812187 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:36:05.812165    4278 scope.go:122] "RemoveContainer" containerID="7133d718c55166c16e8306ff38f550576c9aff7c2c7e74106e3bfc87845becb0"
Oct 30 03:36:15.383435 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:36:15.383390    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:36:19.595252 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:36:19.595208    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:36:24.188159 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:36:24.188118    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:36:36.486255 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:36:36.486214    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:36:41.119363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:36:41.119323    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:36:54.190540 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:36:54.190505    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:36:57.810830 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:36:57.810796    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:37:05.842461 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:37:05.842408    4278 scope.go:122] "RemoveContainer" containerID="2666579ba37c076ef1461972884728a289b26fd676897c53c7b2dabeff1a0cc5"
Oct 30 03:37:05.850870 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:37:05.850843    4278 scope.go:122] "RemoveContainer" containerID="a694b93f458413ae844a18919302f127966562411330a48301ba8d0257d3ea7e"
Oct 30 03:37:05.858401 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:37:05.858381    4278 scope.go:122] "RemoveContainer" containerID="dd97485182d1359a4161f2d62d84921fe2eb3f18d5bcbee11b9dc5c2ca97fa83"
Oct 30 03:37:05.865638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:37:05.865615    4278 scope.go:122] "RemoveContainer" containerID="387e3f2e8c44689e60031354c872b5fef84fa9900c48338e617d7f17712e6307"
Oct 30 03:37:08.674139 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:37:08.674106    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:37:13.596068 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:37:13.596033    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:37:24.191816 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:37:24.191778    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:37:37.034147 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:37:37.034115    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"]
Oct 30 03:37:37.047352 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:37:37.047323    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df53da00-jp797"]
Oct 30 03:37:38.197116 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:37:38.197080    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d888547-2902-4190-890d-ea362e6b8b03" path="/var/lib/kubelet/pods/8d888547-2902-4190-890d-ea362e6b8b03/volumes"
Oct 30 03:37:54.192211 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:37:54.192172    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:38:00.679038 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:38:00.679003    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:38:05.894088 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:38:05.894054    4278 scope.go:122] "RemoveContainer" containerID="a248b7b4c11dc8dfc685fdd564b6ed87e4a6430e4aef3e9133d610139415bf35"
Oct 30 03:38:05.902541 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:38:05.902519    4278 scope.go:122] "RemoveContainer" containerID="23e830dfce2f67ad2decf8b6f1f4cc727fac785f6efbef0c19bbb5df825e1fa4"
Oct 30 03:38:05.909926 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:38:05.909906    4278 scope.go:122] "RemoveContainer" containerID="765562c6a80adfa19aefa814cc3ca944c4df24d6fda0a6948194d0e6b2b03987"
Oct 30 03:38:16.289710 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:38:16.289674    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:38:24.188109 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:38:24.188066    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:38:30.386572 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:38:30.386533    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:38:47.782556 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:38:47.782483    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:38:54.187003 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:38:54.186969    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:39:02.107217 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:39:02.107176    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:39:05.934570 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:39:05.934539    4278 scope.go:122] "RemoveContainer" containerID="f7aebf493c7df7bed1f440e7d264e04c22b61a60ec882e7bb0db4d57697f85df"
Oct 30 03:39:24.187842 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:39:24.187796    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:39:54.188255 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:39:54.188209    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:40:24.188614 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:40:24.188567    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:40:49.284701 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:40:49.284666    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:40:54.188359 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:40:54.188327    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:41:04.676297 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:41:04.676256    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:41:24.187455 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:41:24.187400    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:41:31.080882 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:41:31.080842    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:41:45.288283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:41:45.288204    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:41:54.191242 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:41:54.191200    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:42:00.483082 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:42:00.483045    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:42:14.072673 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:42:14.072633    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:42:24.197190 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:42:24.197142    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:42:27.087458 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:42:27.087394    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:42:41.016911 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:42:41.016877    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:42:54.190081 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:42:54.190044    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:42:55.420315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:42:55.420281    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:43:11.091669 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:43:11.091632    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:43:24.189391 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:43:24.189353    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:43:25.576591 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:43:25.576553    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:43:32.069959 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:43:32.069560    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:43:32.069959 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:43:32.069645    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8bc4f" podUID="15027569-cb39-41f1-9ce2-428a5cf9b784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:43:32.134594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:43:32.134540    4278 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c22dw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:43:32.134918 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:43:32.134869    4278 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c22dw" podUID="1554e25e-0893-4e42-a08d-2c8b1a0445ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:43:32.229160 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:43:32.229126    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:43:41.090955 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:43:41.090917    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:43:45.790510 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:43:45.790468    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:43:54.189743 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:43:54.189700    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:44:03.631360 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:03.631319    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:44:13.608085 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:13.608047    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:44:21.995377 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:21.995339    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:44:24.250316 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:44:24.250271    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:44:28.800692 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.800655    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"]
Oct 30 03:44:28.804499 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.804475    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:28.822540 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.822509    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:44:28.822540 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.822526    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:44:28.822723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.822579    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:44:28.822723 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.822605    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:44:28.824703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.824678    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"]
Oct 30 03:44:28.873125 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.873083    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wctt\" (UniqueName: \"kubernetes.io/projected/1860e868-a891-4989-b46c-36dac396ea56-kube-api-access-8wctt\") pod \"maas-api-revoke-sub-f03629db-dcfsx\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:28.873264 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.873146    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1860e868-a891-4989-b46c-36dac396ea56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dcfsx\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:28.873309 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.873293    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1860e868-a891-4989-b46c-36dac396ea56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dcfsx\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:28.974179 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.974148    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1860e868-a891-4989-b46c-36dac396ea56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dcfsx\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:28.974335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.974267    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1860e868-a891-4989-b46c-36dac396ea56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dcfsx\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:28.974335 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.974302    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wctt\" (UniqueName: \"kubernetes.io/projected/1860e868-a891-4989-b46c-36dac396ea56-kube-api-access-8wctt\") pod \"maas-api-revoke-sub-f03629db-dcfsx\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:28.975001 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.974969    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1860e868-a891-4989-b46c-36dac396ea56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dcfsx\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:28.976329 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.976307    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1860e868-a891-4989-b46c-36dac396ea56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dcfsx\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:28.994645 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:28.994615    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wctt\" (UniqueName: \"kubernetes.io/projected/1860e868-a891-4989-b46c-36dac396ea56-kube-api-access-8wctt\") pod \"maas-api-revoke-sub-f03629db-dcfsx\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:29.117799 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:29.117718    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:29.419845 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:29.419816    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"]
Oct 30 03:44:29.422112 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:44:29.422085    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1860e868_a891_4989_b46c_36dac396ea56.slice/crio-fb7d49ff4f77738d955b171d9931da2e4b0223dbe50cf8123370f24208ce3198 WatchSource:0}: Error finding container fb7d49ff4f77738d955b171d9931da2e4b0223dbe50cf8123370f24208ce3198: Status 404 returned error can't find the container with id fb7d49ff4f77738d955b171d9931da2e4b0223dbe50cf8123370f24208ce3198
Oct 30 03:44:30.384705 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:30.384671    4278 generic.go:356] "Generic (PLEG): container finished" podID="1860e868-a891-4989-b46c-36dac396ea56" containerID="cffb81cae75b3fd3a2474a1ba2af10680d4d0c98ca8f9682df29ffa47b7d790d" exitCode=0
Oct 30 03:44:30.385108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:30.384758    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx" event={"ID":"1860e868-a891-4989-b46c-36dac396ea56","Type":"ContainerDied","Data":"cffb81cae75b3fd3a2474a1ba2af10680d4d0c98ca8f9682df29ffa47b7d790d"}
Oct 30 03:44:30.385108 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:30.384811    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx" event={"ID":"1860e868-a891-4989-b46c-36dac396ea56","Type":"ContainerStarted","Data":"fb7d49ff4f77738d955b171d9931da2e4b0223dbe50cf8123370f24208ce3198"}
Oct 30 03:44:31.535591 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:31.535568    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:31.598132 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:31.598102    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1860e868-a891-4989-b46c-36dac396ea56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1860e868-a891-4989-b46c-36dac396ea56-maas-api-xks-ca\") pod \"1860e868-a891-4989-b46c-36dac396ea56\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") "
Oct 30 03:44:31.598331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:31.598142    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1860e868-a891-4989-b46c-36dac396ea56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1860e868-a891-4989-b46c-36dac396ea56-maas-api-openshift-ca\") pod \"1860e868-a891-4989-b46c-36dac396ea56\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") "
Oct 30 03:44:31.598331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:31.598191    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1860e868-a891-4989-b46c-36dac396ea56-kube-api-access-8wctt\" (UniqueName: \"kubernetes.io/projected/1860e868-a891-4989-b46c-36dac396ea56-kube-api-access-8wctt\") pod \"1860e868-a891-4989-b46c-36dac396ea56\" (UID: \"1860e868-a891-4989-b46c-36dac396ea56\") "
Oct 30 03:44:31.598596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:31.598571    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1860e868-a891-4989-b46c-36dac396ea56-maas-api-openshift-ca" pod "1860e868-a891-4989-b46c-36dac396ea56" (UID: "1860e868-a891-4989-b46c-36dac396ea56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:31.600237 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:31.600211    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1860e868-a891-4989-b46c-36dac396ea56-maas-api-xks-ca" pod "1860e868-a891-4989-b46c-36dac396ea56" (UID: "1860e868-a891-4989-b46c-36dac396ea56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:31.600354 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:31.600333    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1860e868-a891-4989-b46c-36dac396ea56-kube-api-access-8wctt" pod "1860e868-a891-4989-b46c-36dac396ea56" (UID: "1860e868-a891-4989-b46c-36dac396ea56"). InnerVolumeSpecName "kube-api-access-8wctt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:31.699879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:31.699794    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1860e868-a891-4989-b46c-36dac396ea56-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:44:31.699879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:31.699828    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1860e868-a891-4989-b46c-36dac396ea56-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:44:31.699879 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:31.699840    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wctt\" (UniqueName: \"kubernetes.io/projected/1860e868-a891-4989-b46c-36dac396ea56-kube-api-access-8wctt\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:44:32.401160 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:32.401126    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx" event={"ID":"1860e868-a891-4989-b46c-36dac396ea56","Type":"ContainerDied","Data":"fb7d49ff4f77738d955b171d9931da2e4b0223dbe50cf8123370f24208ce3198"}
Oct 30 03:44:32.401322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:32.401174    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb7d49ff4f77738d955b171d9931da2e4b0223dbe50cf8123370f24208ce3198"
Oct 30 03:44:32.401322 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:32.401173    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dcfsx"
Oct 30 03:44:35.403724 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:35.403695    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:44:41.048440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.048390    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"]
Oct 30 03:44:41.048905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.048888    4278 cpu_manager.go:414] "containerMap: removing container" podUID="1860e868-a891-4989-b46c-36dac396ea56" containerName="revoke-keys"
Oct 30 03:44:41.048955 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.048907    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1860e868-a891-4989-b46c-36dac396ea56" containerName="revoke-keys"
Oct 30 03:44:41.049020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.049010    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="1860e868-a891-4989-b46c-36dac396ea56" containerName="revoke-keys"
Oct 30 03:44:41.051991 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.051975    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.089477 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.089411    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:44:41.089702 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.089561    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:44:41.090003 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.089978    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:44:41.090671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.089335    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:44:41.092670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.092648    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"]
Oct 30 03:44:41.231413 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.231381    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7tb7\" (UniqueName: \"kubernetes.io/projected/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-kube-api-access-j7tb7\") pod \"maas-api-revoke-sub-0165d7d2-j8b9l\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.231599 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.231449    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-j8b9l\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.231599 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.231564    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-j8b9l\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.332671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.332599    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-j8b9l\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.332671 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.332665    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7tb7\" (UniqueName: \"kubernetes.io/projected/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-kube-api-access-j7tb7\") pod \"maas-api-revoke-sub-0165d7d2-j8b9l\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.332835 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.332713    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-j8b9l\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.333331 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.333299    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-j8b9l\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.334965 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.334937    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-j8b9l\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.353629 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.353605    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7tb7\" (UniqueName: \"kubernetes.io/projected/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-kube-api-access-j7tb7\") pod \"maas-api-revoke-sub-0165d7d2-j8b9l\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.364314 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.364292    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:41.660315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:41.660289    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"]
Oct 30 03:44:41.661920 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:44:41.661894    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2056f4_ff32_46ce_8457_e3ed5f6c42b7.slice/crio-4b2f6391ad29f4b54c0048a145cec706ccc286858d71652efbeba322ce0178be WatchSource:0}: Error finding container 4b2f6391ad29f4b54c0048a145cec706ccc286858d71652efbeba322ce0178be: Status 404 returned error can't find the container with id 4b2f6391ad29f4b54c0048a145cec706ccc286858d71652efbeba322ce0178be
Oct 30 03:44:42.451753 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:42.451720    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l" event={"ID":"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7","Type":"ContainerStarted","Data":"cec0f4a81ecdf0f1b1229e944b9df97c6169622d9094f0460eac3503e139674c"}
Oct 30 03:44:42.452135 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:42.451770    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l" event={"ID":"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7","Type":"ContainerStarted","Data":"4b2f6391ad29f4b54c0048a145cec706ccc286858d71652efbeba322ce0178be"}
Oct 30 03:44:42.496052 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:42.496000    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l" podStartSLOduration=1.495981697 podStartE2EDuration="1.495981697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:42.495447596 +0000 UTC m=+3820.419655794" watchObservedRunningTime="2026-10-30 03:44:42.495981697 +0000 UTC m=+3820.420189886"
Oct 30 03:44:43.457316 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:43.457282    4278 generic.go:356] "Generic (PLEG): container finished" podID="3b2056f4-ff32-46ce-8457-e3ed5f6c42b7" containerID="cec0f4a81ecdf0f1b1229e944b9df97c6169622d9094f0460eac3503e139674c" exitCode=0
Oct 30 03:44:43.457748 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:43.457360    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l" event={"ID":"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7","Type":"ContainerDied","Data":"cec0f4a81ecdf0f1b1229e944b9df97c6169622d9094f0460eac3503e139674c"}
Oct 30 03:44:44.596905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:44.596882    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:44.762063 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:44.762034    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-kube-api-access-j7tb7\" (UniqueName: \"kubernetes.io/projected/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-kube-api-access-j7tb7\") pod \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") "
Oct 30 03:44:44.762248 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:44.762152    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-openshift-ca\") pod \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") "
Oct 30 03:44:44.762248 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:44.762199    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-xks-ca\") pod \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\" (UID: \"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7\") "
Oct 30 03:44:44.762529 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:44.762504    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-openshift-ca" pod "3b2056f4-ff32-46ce-8457-e3ed5f6c42b7" (UID: "3b2056f4-ff32-46ce-8457-e3ed5f6c42b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:44.764192 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:44.764165    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-xks-ca" pod "3b2056f4-ff32-46ce-8457-e3ed5f6c42b7" (UID: "3b2056f4-ff32-46ce-8457-e3ed5f6c42b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:44.764293 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:44.764248    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-kube-api-access-j7tb7" pod "3b2056f4-ff32-46ce-8457-e3ed5f6c42b7" (UID: "3b2056f4-ff32-46ce-8457-e3ed5f6c42b7"). InnerVolumeSpecName "kube-api-access-j7tb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:44.863484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:44.863460    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.863484 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:44.863482    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.863638 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:44.863493    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7tb7\" (UniqueName: \"kubernetes.io/projected/3b2056f4-ff32-46ce-8457-e3ed5f6c42b7-kube-api-access-j7tb7\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:44:45.467097 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:45.467065    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l"
Oct 30 03:44:45.467097 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:45.467082    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-j8b9l" event={"ID":"3b2056f4-ff32-46ce-8457-e3ed5f6c42b7","Type":"ContainerDied","Data":"4b2f6391ad29f4b54c0048a145cec706ccc286858d71652efbeba322ce0178be"}
Oct 30 03:44:45.467315 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:45.467156    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b2f6391ad29f4b54c0048a145cec706ccc286858d71652efbeba322ce0178be"
Oct 30 03:44:52.020800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.020764    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"]
Oct 30 03:44:52.021214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.021173    4278 cpu_manager.go:414] "containerMap: removing container" podUID="3b2056f4-ff32-46ce-8457-e3ed5f6c42b7" containerName="revoke-keys"
Oct 30 03:44:52.021214 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.021185    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2056f4-ff32-46ce-8457-e3ed5f6c42b7" containerName="revoke-keys"
Oct 30 03:44:52.021304 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.021278    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2056f4-ff32-46ce-8457-e3ed5f6c42b7" containerName="revoke-keys"
Oct 30 03:44:52.024389 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.024368    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.026934 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.026906    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qff4v\" (UniqueName: \"kubernetes.io/projected/dbb7eb04-3783-4d8b-befe-261137acf2e0-kube-api-access-qff4v\") pod \"maas-api-revoke-sub-6b19338f-h44ss\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.027096 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.026950    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-h44ss\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.027096 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.026977    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-h44ss\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.054658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.054634    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:44:52.054823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.054690    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:44:52.054823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.054744    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:44:52.054823 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.054782    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:44:52.056677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.056652    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"]
Oct 30 03:44:52.128369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.128335    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qff4v\" (UniqueName: \"kubernetes.io/projected/dbb7eb04-3783-4d8b-befe-261137acf2e0-kube-api-access-qff4v\") pod \"maas-api-revoke-sub-6b19338f-h44ss\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.128565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.128388    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-h44ss\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.128565 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.128444    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-h44ss\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.129079 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.129052    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-h44ss\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.130670 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.130653    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-h44ss\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.149175 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.149151    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qff4v\" (UniqueName: \"kubernetes.io/projected/dbb7eb04-3783-4d8b-befe-261137acf2e0-kube-api-access-qff4v\") pod \"maas-api-revoke-sub-6b19338f-h44ss\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.337591 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.337526    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:52.636644 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:52.636608    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"]
Oct 30 03:44:52.648288 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:44:52.648254    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb7eb04_3783_4d8b_befe_261137acf2e0.slice/crio-634c161e0119a5f2a9c3c970a6c6c38677e5256aa424c67d56fdeea9aaffffb2 WatchSource:0}: Error finding container 634c161e0119a5f2a9c3c970a6c6c38677e5256aa424c67d56fdeea9aaffffb2: Status 404 returned error can't find the container with id 634c161e0119a5f2a9c3c970a6c6c38677e5256aa424c67d56fdeea9aaffffb2
Oct 30 03:44:53.505574 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:53.505546    4278 generic.go:356] "Generic (PLEG): container finished" podID="dbb7eb04-3783-4d8b-befe-261137acf2e0" containerID="6f8200360f1cde6a30f018596828b5e79a10a9bab3ef71e384a46376254e880f" exitCode=0
Oct 30 03:44:53.505891 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:53.505600    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss" event={"ID":"dbb7eb04-3783-4d8b-befe-261137acf2e0","Type":"ContainerDied","Data":"6f8200360f1cde6a30f018596828b5e79a10a9bab3ef71e384a46376254e880f"}
Oct 30 03:44:53.505891 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:53.505642    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss" event={"ID":"dbb7eb04-3783-4d8b-befe-261137acf2e0","Type":"ContainerStarted","Data":"634c161e0119a5f2a9c3c970a6c6c38677e5256aa424c67d56fdeea9aaffffb2"}
Oct 30 03:44:54.187375 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:44:54.187345    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:44:54.637994 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:54.637970    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:54.653324 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:54.653298    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-openshift-ca\") pod \"dbb7eb04-3783-4d8b-befe-261137acf2e0\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") "
Oct 30 03:44:54.653420 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:54.653396    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-xks-ca\") pod \"dbb7eb04-3783-4d8b-befe-261137acf2e0\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") "
Oct 30 03:44:54.653497 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:54.653440    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb7eb04-3783-4d8b-befe-261137acf2e0-kube-api-access-qff4v\" (UniqueName: \"kubernetes.io/projected/dbb7eb04-3783-4d8b-befe-261137acf2e0-kube-api-access-qff4v\") pod \"dbb7eb04-3783-4d8b-befe-261137acf2e0\" (UID: \"dbb7eb04-3783-4d8b-befe-261137acf2e0\") "
Oct 30 03:44:54.653630 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:54.653608    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-openshift-ca" pod "dbb7eb04-3783-4d8b-befe-261137acf2e0" (UID: "dbb7eb04-3783-4d8b-befe-261137acf2e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:54.653718 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:54.653694    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-openshift-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:44:54.655287 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:54.655266    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-xks-ca" pod "dbb7eb04-3783-4d8b-befe-261137acf2e0" (UID: "dbb7eb04-3783-4d8b-befe-261137acf2e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:54.655416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:54.655399    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb7eb04-3783-4d8b-befe-261137acf2e0-kube-api-access-qff4v" pod "dbb7eb04-3783-4d8b-befe-261137acf2e0" (UID: "dbb7eb04-3783-4d8b-befe-261137acf2e0"). InnerVolumeSpecName "kube-api-access-qff4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:54.755157 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:54.755124    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb7eb04-3783-4d8b-befe-261137acf2e0-maas-api-xks-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:44:54.755157 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:54.755151    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qff4v\" (UniqueName: \"kubernetes.io/projected/dbb7eb04-3783-4d8b-befe-261137acf2e0-kube-api-access-qff4v\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:44:55.516151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:55.516118    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss" event={"ID":"dbb7eb04-3783-4d8b-befe-261137acf2e0","Type":"ContainerDied","Data":"634c161e0119a5f2a9c3c970a6c6c38677e5256aa424c67d56fdeea9aaffffb2"}
Oct 30 03:44:55.516151 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:55.516150    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h44ss"
Oct 30 03:44:55.516355 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:44:55.516173    4278 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="634c161e0119a5f2a9c3c970a6c6c38677e5256aa424c67d56fdeea9aaffffb2"
Oct 30 03:45:00.153981 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.153943    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"]
Oct 30 03:45:00.154486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.154363    4278 cpu_manager.go:414] "containerMap: removing container" podUID="dbb7eb04-3783-4d8b-befe-261137acf2e0" containerName="revoke-keys"
Oct 30 03:45:00.154486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.154374    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb7eb04-3783-4d8b-befe-261137acf2e0" containerName="revoke-keys"
Oct 30 03:45:00.154486 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.154478    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb7eb04-3783-4d8b-befe-261137acf2e0" containerName="revoke-keys"
Oct 30 03:45:00.157631 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.157609    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"
Oct 30 03:45:00.178649 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.178622    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:45:00.187124 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.187095    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:45:00.187306 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.187182    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:45:00.189992 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.189966    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"]
Oct 30 03:45:00.203055 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.203024    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b76804-c93d-45dd-9c1d-978a9c1596b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888865-8zzjl\" (UID: \"a0b76804-c93d-45dd-9c1d-978a9c1596b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"
Oct 30 03:45:00.203194 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.203095    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85f96\" (UniqueName: \"kubernetes.io/projected/a0b76804-c93d-45dd-9c1d-978a9c1596b8-kube-api-access-85f96\") pod \"maas-api-key-cleanup-29888865-8zzjl\" (UID: \"a0b76804-c93d-45dd-9c1d-978a9c1596b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"
Oct 30 03:45:00.304658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.304624    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85f96\" (UniqueName: \"kubernetes.io/projected/a0b76804-c93d-45dd-9c1d-978a9c1596b8-kube-api-access-85f96\") pod \"maas-api-key-cleanup-29888865-8zzjl\" (UID: \"a0b76804-c93d-45dd-9c1d-978a9c1596b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"
Oct 30 03:45:00.304856 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.304693    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b76804-c93d-45dd-9c1d-978a9c1596b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888865-8zzjl\" (UID: \"a0b76804-c93d-45dd-9c1d-978a9c1596b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"
Oct 30 03:45:00.305273 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.305252    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b76804-c93d-45dd-9c1d-978a9c1596b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888865-8zzjl\" (UID: \"a0b76804-c93d-45dd-9c1d-978a9c1596b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"
Oct 30 03:45:00.325925 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.325900    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85f96\" (UniqueName: \"kubernetes.io/projected/a0b76804-c93d-45dd-9c1d-978a9c1596b8-kube-api-access-85f96\") pod \"maas-api-key-cleanup-29888865-8zzjl\" (UID: \"a0b76804-c93d-45dd-9c1d-978a9c1596b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"
Oct 30 03:45:00.470778 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.470705    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"
Oct 30 03:45:00.774577 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:00.774550    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"]
Oct 30 03:45:00.775768 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:45:00.775734    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b76804_c93d_45dd_9c1d_978a9c1596b8.slice/crio-e6d4aa2bdff38c00fd8639e2d770654e9715986e51ceb2b9b20193d7b8a93588 WatchSource:0}: Error finding container e6d4aa2bdff38c00fd8639e2d770654e9715986e51ceb2b9b20193d7b8a93588: Status 404 returned error can't find the container with id e6d4aa2bdff38c00fd8639e2d770654e9715986e51ceb2b9b20193d7b8a93588
Oct 30 03:45:01.545896 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:01.545859    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" event={"ID":"a0b76804-c93d-45dd-9c1d-978a9c1596b8","Type":"ContainerStarted","Data":"13c44c7b885e865a82162b43078ba2dc5a2a75b709da416a8112f83954e6c755"}
Oct 30 03:45:01.546291 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:01.545920    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" event={"ID":"a0b76804-c93d-45dd-9c1d-978a9c1596b8","Type":"ContainerStarted","Data":"e6d4aa2bdff38c00fd8639e2d770654e9715986e51ceb2b9b20193d7b8a93588"}
Oct 30 03:45:01.605722 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:01.605672    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" podStartSLOduration=1.6056578639999999 podStartE2EDuration="1.605657864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:01.605356343 +0000 UTC m=+3839.529564530" watchObservedRunningTime="2026-10-30 03:45:01.605657864 +0000 UTC m=+3839.529866050"
Oct 30 03:45:02.552198 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:02.552164    4278 generic.go:356] "Generic (PLEG): container finished" podID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" containerID="13c44c7b885e865a82162b43078ba2dc5a2a75b709da416a8112f83954e6c755" exitCode=60
Oct 30 03:45:02.552624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:02.552256    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" event={"ID":"a0b76804-c93d-45dd-9c1d-978a9c1596b8","Type":"ContainerDied","Data":"13c44c7b885e865a82162b43078ba2dc5a2a75b709da416a8112f83954e6c755"}
Oct 30 03:45:02.552624 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:02.552504    4278 scope.go:122] "RemoveContainer" containerID="13c44c7b885e865a82162b43078ba2dc5a2a75b709da416a8112f83954e6c755"
Oct 30 03:45:03.295118 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:03.295082    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vx95s"]
Oct 30 03:45:03.559020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:03.558929    4278 generic.go:356] "Generic (PLEG): container finished" podID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" containerID="719a545b9664fe6d03a1946d58ce82789ba66fa74c40ddbd0c7acb1138dcfb69" exitCode=60
Oct 30 03:45:03.559020 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:03.558969    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" event={"ID":"a0b76804-c93d-45dd-9c1d-978a9c1596b8","Type":"ContainerDied","Data":"719a545b9664fe6d03a1946d58ce82789ba66fa74c40ddbd0c7acb1138dcfb69"}
Oct 30 03:45:03.559542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:03.559026    4278 scope.go:122] "RemoveContainer" containerID="13c44c7b885e865a82162b43078ba2dc5a2a75b709da416a8112f83954e6c755"
Oct 30 03:45:03.559542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:03.559248    4278 scope.go:122] "RemoveContainer" containerID="719a545b9664fe6d03a1946d58ce82789ba66fa74c40ddbd0c7acb1138dcfb69"
Oct 30 03:45:03.559651 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:45:03.559586    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888865-8zzjl_odh-ai-gateway-infra(a0b76804-c93d-45dd-9c1d-978a9c1596b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8"
Oct 30 03:45:04.606449 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:04.588105    4278 scope.go:122] "RemoveContainer" containerID="719a545b9664fe6d03a1946d58ce82789ba66fa74c40ddbd0c7acb1138dcfb69"
Oct 30 03:45:04.606449 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:45:04.588396    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888865-8zzjl_odh-ai-gateway-infra(a0b76804-c93d-45dd-9c1d-978a9c1596b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8"
Oct 30 03:45:14.361221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:14.361191    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rbqcv_d1772e43-22c4-4f22-b106-704810d9d9cb/manager/0.log"
Oct 30 03:45:14.796903 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:14.796872    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-474bl_f6d27d9d-a11d-4f43-9c83-02658f4b0079/manager/0.log"
Oct 30 03:45:15.276968 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:15.276928    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hhdnz_6f0f3468-47c5-489c-9a4c-bf3279e50bdc/server/0.log"
Oct 30 03:45:15.474511 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:15.474478    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-7qqcm_82bfdb57-cc86-4f29-aa87-43a3c3682105/manager/0.log"
Oct 30 03:45:15.695168 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:15.695098    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-95rk5_429df927-d84d-4d8c-9111-263b24400651/manager/0.log"
Oct 30 03:45:16.204222 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:16.204195    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-b7ftg_1c53f0f1-2c25-41cc-8049-a17305d2dc2c/copy-manifests/0.log"
Oct 30 03:45:16.235090 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:16.235060    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-b7ftg_1c53f0f1-2c25-41cc-8049-a17305d2dc2c/manager/0.log"
Oct 30 03:45:17.851520 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:17.851487    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29888865-8zzjl_a0b76804-c93d-45dd-9c1d-978a9c1596b8/cleanup/1.log"
Oct 30 03:45:18.040375 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:18.040346    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-j8b9l_3b2056f4-ff32-46ce-8457-e3ed5f6c42b7/revoke-keys/0.log"
Oct 30 03:45:18.192080 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:18.191991    4278 scope.go:122] "RemoveContainer" containerID="719a545b9664fe6d03a1946d58ce82789ba66fa74c40ddbd0c7acb1138dcfb69"
Oct 30 03:45:18.609569 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:18.609541    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-h44ss_dbb7eb04-3783-4d8b-befe-261137acf2e0/revoke-keys/0.log"
Oct 30 03:45:18.653389 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:18.653357    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" event={"ID":"a0b76804-c93d-45dd-9c1d-978a9c1596b8","Type":"ContainerStarted","Data":"1bc32906c6be6422e74bcdfa366e6e3131a58951ef70077def3b94bf7ead38c8"}
Oct 30 03:45:19.377905 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:19.377874    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-dcfsx_1860e868-a891-4989-b46c-36dac396ea56/revoke-keys/0.log"
Oct 30 03:45:19.659395 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:19.659315    4278 generic.go:356] "Generic (PLEG): container finished" podID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" containerID="1bc32906c6be6422e74bcdfa366e6e3131a58951ef70077def3b94bf7ead38c8" exitCode=60
Oct 30 03:45:19.659613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:19.659393    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" event={"ID":"a0b76804-c93d-45dd-9c1d-978a9c1596b8","Type":"ContainerDied","Data":"1bc32906c6be6422e74bcdfa366e6e3131a58951ef70077def3b94bf7ead38c8"}
Oct 30 03:45:19.659613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:19.659473    4278 scope.go:122] "RemoveContainer" containerID="719a545b9664fe6d03a1946d58ce82789ba66fa74c40ddbd0c7acb1138dcfb69"
Oct 30 03:45:19.659771 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:19.659753    4278 scope.go:122] "RemoveContainer" containerID="1bc32906c6be6422e74bcdfa366e6e3131a58951ef70077def3b94bf7ead38c8"
Oct 30 03:45:19.659991 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:45:19.659973    4278 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888865-8zzjl_odh-ai-gateway-infra(a0b76804-c93d-45dd-9c1d-978a9c1596b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8"
Oct 30 03:45:19.714276 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:19.714241    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"]
Oct 30 03:45:20.171418 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:45:20.171381    4278 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719a545b9664fe6d03a1946d58ce82789ba66fa74c40ddbd0c7acb1138dcfb69\": container with ID starting with 719a545b9664fe6d03a1946d58ce82789ba66fa74c40ddbd0c7acb1138dcfb69 not found: ID does not exist" containerID="719a545b9664fe6d03a1946d58ce82789ba66fa74c40ddbd0c7acb1138dcfb69"
Oct 30 03:45:20.812241 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:20.812214    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"
Oct 30 03:45:20.899703 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:20.899666    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b76804-c93d-45dd-9c1d-978a9c1596b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b76804-c93d-45dd-9c1d-978a9c1596b8-maas-api-service-ca\") pod \"a0b76804-c93d-45dd-9c1d-978a9c1596b8\" (UID: \"a0b76804-c93d-45dd-9c1d-978a9c1596b8\") "
Oct 30 03:45:20.899904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:20.899716    4278 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b76804-c93d-45dd-9c1d-978a9c1596b8-kube-api-access-85f96\" (UniqueName: \"kubernetes.io/projected/a0b76804-c93d-45dd-9c1d-978a9c1596b8-kube-api-access-85f96\") pod \"a0b76804-c93d-45dd-9c1d-978a9c1596b8\" (UID: \"a0b76804-c93d-45dd-9c1d-978a9c1596b8\") "
Oct 30 03:45:20.900086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:20.900062    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b76804-c93d-45dd-9c1d-978a9c1596b8-maas-api-service-ca" pod "a0b76804-c93d-45dd-9c1d-978a9c1596b8" (UID: "a0b76804-c93d-45dd-9c1d-978a9c1596b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:20.901851 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:20.901817    4278 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b76804-c93d-45dd-9c1d-978a9c1596b8-kube-api-access-85f96" pod "a0b76804-c93d-45dd-9c1d-978a9c1596b8" (UID: "a0b76804-c93d-45dd-9c1d-978a9c1596b8"). InnerVolumeSpecName "kube-api-access-85f96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:21.000525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:21.000448    4278 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b76804-c93d-45dd-9c1d-978a9c1596b8-maas-api-service-ca\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.000525 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:21.000475    4278 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85f96\" (UniqueName: \"kubernetes.io/projected/a0b76804-c93d-45dd-9c1d-978a9c1596b8-kube-api-access-85f96\") on node \"ip-10-0-128-214.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.671639 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:21.671609    4278 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"
Oct 30 03:45:21.671881 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:21.671642    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl" event={"ID":"a0b76804-c93d-45dd-9c1d-978a9c1596b8","Type":"ContainerDied","Data":"e6d4aa2bdff38c00fd8639e2d770654e9715986e51ceb2b9b20193d7b8a93588"}
Oct 30 03:45:21.671881 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:21.671701    4278 scope.go:122] "RemoveContainer" containerID="1bc32906c6be6422e74bcdfa366e6e3131a58951ef70077def3b94bf7ead38c8"
Oct 30 03:45:21.725363 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:21.725331    4278 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"]
Oct 30 03:45:21.738509 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:21.738479    4278 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-8zzjl"]
Oct 30 03:45:22.197517 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:22.197478    4278 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" path="/var/lib/kubelet/pods/a0b76804-c93d-45dd-9c1d-978a9c1596b8/volumes"
Oct 30 03:45:22.580371 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:22.580341    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vx95s_9617029d-abc2-416a-9628-0e25453cafe7/limitador/0.log"
Oct 30 03:45:22.785338 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:22.785307    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-xv2td_b28fd796-8c22-49dd-92d4-714303b3ae5e/manager/0.log"
Oct 30 03:45:23.594806 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:23.594771    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-tkhp2_f1ac3110-5846-48f8-a0d4-edf87b4defb1/istio-proxy/0.log"
Oct 30 03:45:24.188398 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:45:24.188366    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:45:24.874097 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:24.874064    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-8bc4f_15027569-cb39-41f1-9ce2-428a5cf9b784/istio-proxy/0.log"
Oct 30 03:45:25.091899 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:25.091864    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-c22dw_1554e25e-0893-4e42-a08d-2c8b1a0445ba/istio-proxy/0.log"
Oct 30 03:45:26.124281 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:26.124248    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-x5p4f_7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae/istio-proxy/0.log"
Oct 30 03:45:39.029221 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:39.029187    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rbqcv_d1772e43-22c4-4f22-b106-704810d9d9cb/manager/0.log"
Oct 30 03:45:39.450684 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:39.450600    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-474bl_f6d27d9d-a11d-4f43-9c83-02658f4b0079/manager/0.log"
Oct 30 03:45:39.881586 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:39.881554    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hhdnz_6f0f3468-47c5-489c-9a4c-bf3279e50bdc/server/0.log"
Oct 30 03:45:40.082216 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:40.082182    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-7qqcm_82bfdb57-cc86-4f29-aa87-43a3c3682105/manager/0.log"
Oct 30 03:45:40.306850 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:40.306801    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-95rk5_429df927-d84d-4d8c-9111-263b24400651/manager/0.log"
Oct 30 03:45:40.821613 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:40.821583    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-b7ftg_1c53f0f1-2c25-41cc-8049-a17305d2dc2c/copy-manifests/0.log"
Oct 30 03:45:40.852952 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:40.852920    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-b7ftg_1c53f0f1-2c25-41cc-8049-a17305d2dc2c/manager/0.log"
Oct 30 03:45:42.230283 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:42.230253    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-j8b9l_3b2056f4-ff32-46ce-8457-e3ed5f6c42b7/revoke-keys/0.log"
Oct 30 03:45:42.795923 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:42.795897    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-h44ss_dbb7eb04-3783-4d8b-befe-261137acf2e0/revoke-keys/0.log"
Oct 30 03:45:43.568115 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:43.568085    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-dcfsx_1860e868-a891-4989-b46c-36dac396ea56/revoke-keys/0.log"
Oct 30 03:45:46.118818 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:46.118783    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vx95s_9617029d-abc2-416a-9628-0e25453cafe7/limitador/0.log"
Oct 30 03:45:46.321690 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:46.321658    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-xv2td_b28fd796-8c22-49dd-92d4-714303b3ae5e/manager/0.log"
Oct 30 03:45:46.884472 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:46.884439    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-tkhp2_f1ac3110-5846-48f8-a0d4-edf87b4defb1/istio-proxy/0.log"
Oct 30 03:45:48.046409 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:48.046377    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-8bc4f_15027569-cb39-41f1-9ce2-428a5cf9b784/istio-proxy/0.log"
Oct 30 03:45:48.238369 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:48.238334    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-c22dw_1554e25e-0893-4e42-a08d-2c8b1a0445ba/istio-proxy/0.log"
Oct 30 03:45:49.230416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:45:49.230385    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-x5p4f_7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae/istio-proxy/0.log"
Oct 30 03:45:54.186964 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:45:54.186925    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:46:24.186550 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:46:24.186506    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:46:54.187694 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:46:54.187658    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:47:10.801557 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:10.801521    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wttwd_8308710a-3726-4a5e-90b5-7a6d63a1a25b/global-pull-secret-syncer/0.log"
Oct 30 03:47:11.584731 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:11.584700    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-crnh2_729ae77b-bc29-4685-b33b-38feb7fe0a68/konnectivity-agent/0.log"
Oct 30 03:47:11.682308 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:11.682275    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-214.ec2.internal_4b5a364f76591020026198c5396db417/haproxy/0.log"
Oct 30 03:47:19.266680 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:19.266642    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vx95s_9617029d-abc2-416a-9628-0e25453cafe7/limitador/0.log"
Oct 30 03:47:19.491730 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:19.491694    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-xv2td_b28fd796-8c22-49dd-92d4-714303b3ae5e/manager/0.log"
Oct 30 03:47:20.996885 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:20.996849    4278 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"]
Oct 30 03:47:20.997326 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:20.997309    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" containerName="cleanup"
Oct 30 03:47:20.997417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:20.997341    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" containerName="cleanup"
Oct 30 03:47:20.997417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:20.997379    4278 cpu_manager.go:414] "containerMap: removing container" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" containerName="cleanup"
Oct 30 03:47:20.997417 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:20.997385    4278 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" containerName="cleanup"
Oct 30 03:47:20.997596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:20.997474    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" containerName="cleanup"
Oct 30 03:47:20.997596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:20.997490    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" containerName="cleanup"
Oct 30 03:47:20.997596 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:20.997498    4278 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b76804-c93d-45dd-9c1d-978a9c1596b8" containerName="cleanup"
Oct 30 03:47:21.000658 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.000614    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.016904 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.016883    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zst6n\"/\"kube-root-ca.crt\""
Oct 30 03:47:21.019677 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.019652    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"]
Oct 30 03:47:21.030077 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.030051    4278 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zst6n\"/\"openshift-service-ca.crt\""
Oct 30 03:47:21.030218 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.030171    4278 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zst6n\"/\"default-dockercfg-jx5vb\""
Oct 30 03:47:21.121766 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.121725    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-podres\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.121951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.121842    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-sys\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.121951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.121895    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-proc\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.121951 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.121926    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-lib-modules\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.122069 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.121953    4278 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrv8\" (UniqueName: \"kubernetes.io/projected/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-kube-api-access-5vrv8\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.223414 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.223378    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-lib-modules\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.223643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.223453    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrv8\" (UniqueName: \"kubernetes.io/projected/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-kube-api-access-5vrv8\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.223643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.223525    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-podres\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.223643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.223569    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-podres\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.223643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.223527    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-lib-modules\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.223643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.223596    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-sys\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.223643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.223627    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-sys\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.223643 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.223640    4278 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-proc\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.223954 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.223700    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-proc\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.244114 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.244082    4278 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrv8\" (UniqueName: \"kubernetes.io/projected/6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9-kube-api-access-5vrv8\") pod \"perf-node-gather-daemonset-zftbg\" (UID: \"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.313901 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.313871    4278 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:21.607376 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.607350    4278 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"]
Oct 30 03:47:21.609464 ip-10-0-128-214 kubenswrapper[4278]: W1030 03:47:21.609436    4278 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6033e97b_e5a5_4c99_bc6f_99b3b70b0bb9.slice/crio-aaf152466f86e9a3529dd522674d69c44e5d96fb8f6af70f864de32a3cdd51fc WatchSource:0}: Error finding container aaf152466f86e9a3529dd522674d69c44e5d96fb8f6af70f864de32a3cdd51fc: Status 404 returned error can't find the container with id aaf152466f86e9a3529dd522674d69c44e5d96fb8f6af70f864de32a3cdd51fc
Oct 30 03:47:21.766005 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.765975    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bhd9g_12f3a6be-b590-4482-b69c-2429c37d0dff/kube-state-metrics/0.log"
Oct 30 03:47:21.866248 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.866167    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bhd9g_12f3a6be-b590-4482-b69c-2429c37d0dff/kube-rbac-proxy-main/0.log"
Oct 30 03:47:21.975066 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:21.975035    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bhd9g_12f3a6be-b590-4482-b69c-2429c37d0dff/kube-rbac-proxy-self/0.log"
Oct 30 03:47:22.232740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:22.232649    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg" event={"ID":"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9","Type":"ContainerStarted","Data":"a2d0ad772e3a3a6d8d551a690770100006c08eec39a9da993a1ee24c7e1ab695"}
Oct 30 03:47:22.232740 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:22.232701    4278 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg" event={"ID":"6033e97b-e5a5-4c99-bc6f-99b3b70b0bb9","Type":"ContainerStarted","Data":"aaf152466f86e9a3529dd522674d69c44e5d96fb8f6af70f864de32a3cdd51fc"}
Oct 30 03:47:22.233186 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:22.232797    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:22.274086 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:22.274034    4278 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg" podStartSLOduration=2.274019194 podStartE2EDuration="2.274019194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:22.272293118 +0000 UTC m=+3980.196501328" watchObservedRunningTime="2026-10-30 03:47:22.274019194 +0000 UTC m=+3980.198227382"
Oct 30 03:47:22.335407 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:22.335374    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-25bsh_c803dbce-4c1c-4d64-a85c-fc766855d54a/node-exporter/0.log"
Oct 30 03:47:22.445203 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:22.445174    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-25bsh_c803dbce-4c1c-4d64-a85c-fc766855d54a/kube-rbac-proxy/0.log"
Oct 30 03:47:22.555728 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:22.555698    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-25bsh_c803dbce-4c1c-4d64-a85c-fc766855d54a/init-textfile/0.log"
Oct 30 03:47:23.355664 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:23.355623    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-k574r_435fb283-f5e3-4a4c-85a1-f64ab00e90b3/kube-rbac-proxy-main/0.log"
Oct 30 03:47:23.465518 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:23.465481    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-k574r_435fb283-f5e3-4a4c-85a1-f64ab00e90b3/kube-rbac-proxy-self/0.log"
Oct 30 03:47:23.575444 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:23.575385    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-k574r_435fb283-f5e3-4a4c-85a1-f64ab00e90b3/openshift-state-metrics/0.log"
Oct 30 03:47:23.698051 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:23.697965    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f7aea3a-e14a-4eec-8d24-41d4d56a8c84/prometheus/0.log"
Oct 30 03:47:23.807381 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:23.807282    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f7aea3a-e14a-4eec-8d24-41d4d56a8c84/config-reloader/0.log"
Oct 30 03:47:23.917871 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:23.917841    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f7aea3a-e14a-4eec-8d24-41d4d56a8c84/thanos-sidecar/0.log"
Oct 30 03:47:24.027542 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:24.027513    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f7aea3a-e14a-4eec-8d24-41d4d56a8c84/kube-rbac-proxy-web/0.log"
Oct 30 03:47:24.138435 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:24.138389    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f7aea3a-e14a-4eec-8d24-41d4d56a8c84/kube-rbac-proxy/0.log"
Oct 30 03:47:24.188704 ip-10-0-128-214 kubenswrapper[4278]: E1030 03:47:24.188670    4278 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs2sf" containerName="extractor"
Oct 30 03:47:24.248318 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:24.248289    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f7aea3a-e14a-4eec-8d24-41d4d56a8c84/kube-rbac-proxy-thanos/0.log"
Oct 30 03:47:24.360419 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:24.360338    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f7aea3a-e14a-4eec-8d24-41d4d56a8c84/init-config-reloader/0.log"
Oct 30 03:47:24.707454 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:24.707363    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-mjczd_f68b6a91-9c7c-4628-87ac-3c681f70c7e0/prometheus-operator-admission-webhook/0.log"
Oct 30 03:47:24.838927 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:24.838894    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bcf4486-xh5ht_ebbb58b1-718b-444d-a2ed-bf7cfb7950c2/telemeter-client/0.log"
Oct 30 03:47:24.961824 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:24.961731    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bcf4486-xh5ht_ebbb58b1-718b-444d-a2ed-bf7cfb7950c2/reload/0.log"
Oct 30 03:47:25.070784 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:25.070753    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bcf4486-xh5ht_ebbb58b1-718b-444d-a2ed-bf7cfb7950c2/kube-rbac-proxy/0.log"
Oct 30 03:47:25.192594 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:25.192558    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675f749d8-tlnp5_5bc64762-491a-420d-aabe-3c8e280bf1ea/thanos-query/0.log"
Oct 30 03:47:25.301675 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:25.301646    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675f749d8-tlnp5_5bc64762-491a-420d-aabe-3c8e280bf1ea/kube-rbac-proxy-web/0.log"
Oct 30 03:47:25.412672 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:25.412633    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675f749d8-tlnp5_5bc64762-491a-420d-aabe-3c8e280bf1ea/kube-rbac-proxy/0.log"
Oct 30 03:47:25.522764 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:25.522732    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675f749d8-tlnp5_5bc64762-491a-420d-aabe-3c8e280bf1ea/prom-label-proxy/0.log"
Oct 30 03:47:25.635266 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:25.635183    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675f749d8-tlnp5_5bc64762-491a-420d-aabe-3c8e280bf1ea/kube-rbac-proxy-rules/0.log"
Oct 30 03:47:25.746261 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:25.746230    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675f749d8-tlnp5_5bc64762-491a-420d-aabe-3c8e280bf1ea/kube-rbac-proxy-metrics/0.log"
Oct 30 03:47:28.247440 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:28.247406    4278 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-zftbg"
Oct 30 03:47:29.015559 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:29.015525    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8454899578-4xqff_3ad111a8-76cf-488a-821a-8c456abc141c/console/0.log"
Oct 30 03:47:29.150986 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:29.150953    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-g6q8h_9c16167c-eef6-4dc3-85e7-d4a419212e43/download-server/0.log"
Oct 30 03:47:32.789205 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:32.789171    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmqlv_7aeb9190-4121-4fd8-a52d-5a270e004d07/dns/0.log"
Oct 30 03:47:32.891248 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:32.891215    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmqlv_7aeb9190-4121-4fd8-a52d-5a270e004d07/kube-rbac-proxy/0.log"
Oct 30 03:47:33.237368 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:33.237280    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-76z2q_01973b42-4064-4c7c-b6ef-b85d89b2f152/dns-node-resolver/0.log"
Oct 30 03:47:33.770380 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:33.770349    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-tkhp2_f1ac3110-5846-48f8-a0d4-edf87b4defb1/istio-proxy/0.log"
Oct 30 03:47:34.748416 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:34.748384    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-28mns_54cc321a-a26d-4aab-8e95-144a2f9aaa41/node-ca/0.log"
Oct 30 03:47:35.898652 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:35.898614    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-8bc4f_15027569-cb39-41f1-9ce2-428a5cf9b784/istio-proxy/0.log"
Oct 30 03:47:35.958800 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:35.958769    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-c22dw_1554e25e-0893-4e42-a08d-2c8b1a0445ba/istio-proxy/0.log"
Oct 30 03:47:36.338537 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:36.338501    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-x5p4f_7f6b6b3b-6b19-46b6-8d0a-a3e4b6d156ae/istio-proxy/0.log"
Oct 30 03:47:36.720204 ip-10-0-128-214 kubenswrapper[4278]: I1030 03:47:36.720121    4278 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-tkhp2_f1ac3110-5846-48f8-a0d4-edf87b4defb1/istio-proxy/0.log"